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FOREWORD 


The  purpose  of  this  handbook  is  to  provide  a  text  and  reference  material 
in  System  Analysis  and  Cost-Effectiveness.  It  is  intended  for  those  technical, 
scientific,  management,  and  administrative  personnel  who  are  responsible  for 
preparing  information,  making  decisions  or  reviewing  decisions  made  by  others 
regarding  life-cycle  cost,  system  effectiveness  (availability,  dependability, 
capability),  or  technical  feasibility  of  a  system  or  equipment  at  any  phase 
in  its  life  cycle.  It  is  immediately  useful  to  personnel  who  are  familiar 
with  a  system  or  equipment  under  study  but  are  not  familiar  with  the  method¬ 
ology  and  techniques  of  System  Analysis  and  Cost-Effectiveness. 

The  handbook  consists  of  four  chapters:  (1)  an  introduction  to  the  con¬ 
cept  of  system  analysis  and  cost-effectiveness;  (2)  a  basic  framework,  or 
general  methodological  approach,  for  conducting  and  reviewing  cost-effectiveness 
or  system  analysis  studies;  (3)  a  set  of  techniques  (linear  programming,  queue¬ 
ing  theory,  simulation,  etc.)  that  can  be  used  for  performing  cost- effectiveness 
and  system  analysis  studies;  and  (4)  a  review  of  the  basic  mathematical  and  sta¬ 
tistical  concepts  that  underlie  the  scientific  approach  in  the  system  analysis/ 
cost-effectiveness  process. 

The  handbook  was  originally  written  by  AR1NC  Research  Corporation,  2551  Riva 
Road,  Annapolis,  Maryland  21401,  in  response  to  line  item  0003  Exhibit  A002  of 
Contract  Number  DAAB07-68-C-0056  for  the  Systems/Cost  Analysis  Office,  U.  S.  Army 
Electronics  Command,  Fort  Monmouth,  New  Jersey  07703.  Messrs.  J.  A.  Macinko 
and  R.  J.  Sanford  were  the  USAECOM  Project  Engineers  and  Mr.  D.  P.  Salvano.  Chief, 
Systems  Evaluation  Division,  was  the  Project  Advisor.  It  is  now  being  published 
as  an  AMC  handbook  in  this  series  designated  AMCP  706-191. 

The  handbooks  are  readily  available  to  all  elements  of  AMC  including  person¬ 
nel  and  contractors  who  have  a  need  and/or  requirement.  The  U.S.  Army  Materiel  Com¬ 
mand  policy  is  to  release  these  Engineering  Design  Handbooks  to  other  DOD  activi¬ 
ties  and  their  contractors,  and  other  Government  agencies,  in  accordance  with 
current  Army  Regulation  70-31,  9  September  1966.  Procedures  for  acquiring  these 
handbooks  follow: 

a.  Activities  within  AMC  and  other  DOD  agencies  should  direct  their  requests 
on  an  official  form  to: 


Commanding  Officer 
Letterkenny  Army  Depot 
ATTN:  AMXLE-ATD 

Chambersburg,  Pennsylvania  17201 
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b  Contractors  who  have  Department  of  Defense  contracts  should  submit 
their  requests,  through  their  contracting  officer  with  proper  justification, 
to  the  address  indicated  in  paragraph  a. 

c.  Government  agencies  other  than  DOD  having  need  for  the  handbooks 
may  submit  their  request  directly  to  the  Letterkenny  Army  Depot,  as  indicated 
in  paragraph  a,  or  to: 


Commanding  General 
U.  S.  Army  Materiel  Command 
ATTN:  AMCAM-ABS 
Washington,  D.  C.  20315 

d.  Industries  not  having  a  Government  contract  (this  includes  Universities) 
must  forward  their  requests  to: 

Commanding  General 
U.  S.  Army  Materiel  Command 
ATTN:  AMCRD-TV 

Washington,  D.  C.  20315 

e.  All  foreign  requests  must  be  submitted  through  the  Washington,  D.  C. 
Embassy  to: 

Assistant  Chief  of  Staff  for  Intelligence 
Foreign  Liaison  Office 
Department  of  the  Army 
Washington,  D.  C.  20310 

All  requests,  other  than  those  originating  within  the  DOD,  must  be  accom¬ 
panied  by  a  valid  justification. 

Comments  and  suggestions  on  this  handbook  are  welcome  and  should  be 
addressed  to  Army  Research  Office-Durham.  Box  CM,  Duke  Station,  Durham,  North 
Carolina  27706. 
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CHAPTER  1 

INTRODUCTION 


1.1  DEFINITIONS  OF  SYSTEMS  ANALYSIS 

Generally  speaking,  the  nomenclature  of  Systems  Analysis  can  be  applied  to 
any  systematic  approach  that  compares  alternate  means  of  attaining  a  specified 
objective.  The  specific  techniques  and  methodologies  may  differ  depending  on 
the  many  factors  of  each  study;  those  inherent  due  to  the  class  of  problem 
and  those  imputed  because  of  problem  variation  from  a  "classic  case".  However, 
all  of  the  generic  classes  of  Systems  Analysis  studies  have  the  common  feature 
of  systematically  examining  all  classes  of  problems,  whether  simple  or  complex. 
The  application  of  System  Analysis  processes  are  directed  towards  supplying 
the  decision-makers  with  maximum  information,  quantified  when  possible,  in 
order  to  help  them  in  selecting  preferred  alternatives  to  the  attainment  of 
the  stated  objective.  Also,  when  no  alternative  means  are  clearly  visible, 
the  process  is  capable  of  imparting  cogent  information  which  can  be  utilized  in 
the  formulation  of  new  alternatives. 

The  concept  of  Systems  Analysis  has  received  considerable  attention  through¬ 
out  Department  of  Defense  areas  of  interest;  Army,  Navy,  Marine  Corp,  and  Air 
Force.  However,  the  subject  and  applicability  are  not  exclusively  military 
oriented.  Extensive  use  of  Systems  Analysis  has  been  utilized  by  non-military 
activities,  beth  in-house  governmental  agencies  as  well  as  the  private  sector 
of  the  economy. 

Materiel  Systems  Analysis  has  been  defined  by  the  United  States  Army 
Materiel  Command  as  follows:  1 

1.  Materiel  Systems  Analysis  -  A  generic  term  which  implies  both  a 
technique  and  a  function  which,  for  the  purposes  of  this  regulation, 
are  defined  as  follows: 

a.  As  a  technique  —  involves  the  analytic  investigation  and  quantita¬ 
tive  appraisal  and  comparison  of  materiel  programs  or  courses  of  action  in 
terms  of  the  effectiveness,  improvement  coefficient  or  cost  benefit  expected 
versus  the  costs  either  required  or  anticipated  to  be  incurred.  Generally, 

AMCR  11-1;  Research  and  Development  Materiel  Systems  Analysis;  U.  S.  Army 
Materiel  Command,  Headquarters,  Washington,  D.  C.  ,  21  April  1970. 
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for  Systems  Analysis  for  materiel  items  or  programs,  the  benefits  and  costs 
of  concern  are  considered  on  a  "life  cycle"  basis.  Systems  Analysis,  as 
a  technique,  may  be  applied  at  any  point  in  the  life  cycle. 

b.  As  a  function  --  involves  the  staff  and  operation  activity  necessary 
and  required  to  discharge  the  AMC  requirement  and  responsibilities  for  Sys¬ 
tems  Analysis  in  an  organized  fashion  and  to  fix  responsibility.  In  general, 
the  conduct  of  the  Systems  Analysis  function  takes  the  form  of  studies, 
projects,  and  investigations  involving  the  technique  described  above  and 
the  application  of  modern  analytics  and- costing  procedures .  The  studies, 
projects,  and  investigations  comprising  the  function  of  Systems  Analysis 
may  variously  take  the  form  and  title  of  cost-effectiveness,  parametric 
design/oost-eff ectiveness  (PD/CE),  cost-benefit,  cost  and  performance, 
trade-off,  optimum  mix,  and  quantitative  inventory  mix  studies  and  analy¬ 
ses;  product- improvement  determinations;  and  qualitative  assessments  of 
approaches  in  functional  activities  and  programs.  The  techniques  of  Sys¬ 
tems  Analysis  are  equally  applicable  to  all  of  the  above.  As  a  function, 
Systems  Analysis  seeks  to  aid  the  decision  making  process  throughout  the 
life  cycle  of  materiel  programs. 

1.2  DEFINITION  OF  COST-EFFECTIVENESS 

1.2.1  Cost-Effectiveness  Analysis  (study)  has  been  defined  by  the  United  States 

p 

Army  Materiel  Command  as  follows: 

"Cost-effectiveness  analysis  (study)  -  The  process  of  comparing  alternative 
solutions  to  mission  requirements  in  terms  of  value  received  (effective¬ 
ness)  for  the  resources  expended  (costs)" 

1.2.2  Cost-ef fectiveness , (C-E)  in  generic  usage,  is  interpreted  as  a  measure 
defined  implicitly  or  explicitly  by  a  decision-maker  of  the  benefits  to  be 
derived  from  and  the  resources  expended  on  a  system ' 

This  can  be  functionally  expressed  as: 

C-E  =  f  (benefits  derived;  resources  expended) 

1.3  BACKGROUND  AND  HISTORY  OF  SYSTEMS  ANALYSIS  AND  COST-EFFECTIVENESS 

1.3.1  History  of  Systems  Analysis 

Present  day  use  of  the  word  "Systems  Analysis"  is  varied,  depending  on  the 
user.  The  chronology  of  its  constituent  elements  could  (at  least)  regress  to: 

~2 - 

Ibid 

'Maltese,  Jasper;  ARINC  Research  Monograph  No.  12.,  System  Cost-Effectiveness; 
Basic  Concepts  and  Framework  for  Analysis  -  ARINC  Research  Corp.,  Annapolis, 

Mi.,  January  1967)  P-9- 
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Aristotelian  logic;  then  to  the  formulation  of  methods  and  procedures  of  science 
during  the  Renaissance  (l4th-17th  centuries);  Fredrick  W.  Taylor's  inception  of 
Scientific  Management;'  sporadic  use  of  statistical  decision  making  in  certain 
World  War  I  studies^  and  introduction  of  a  scientific  method  consisting  of 
objectives,  constraints,  configuration,  selection,  implementation,  evaluation, 
feedback  and  conclusion  -  known  by  college  students  for  years  as  a  format  for 
problem  solving. 

The  nearest  historical  milestone  (within  the  generic  context  of  Systems 
Analysis)  that  has  major  import  to  the  ultimate  definition  is  the  development 
and  use  of  operations  research  in  Great  Britain  during  World  War  II.  These 
operations  research  studies  were  devoted  to  early  warning  systems,  anti-aircraft 
gunnery,  anti-submarine  warfare,  civilian  defense  and  conduct  of  bombing  raids. 

A  group  consisting  of  Professor  P.MS.  Blockett,  three  physiologists,  two 
mathematical  physicists,  one  astro  physicist,  an  Army  officer,  one  surveyor, 
a  general  physicist,  and  two  mathematicians  utilized  the  mixed-group  approach 
in  solving  operational  problems.^  This  philosophy  is  certainly  inherent  in 
what  we  now  call  Systems  Analysis,  with  the  inter-disciplinary  group  being 
necessitated  by  both  the  complexity  of  the  problem  and  its  means  of  solution. 

The  main  difficulty  in  describing  what  "Systems  Analysis"  is  and  is  not 
can  be  gleaned  from  the  newly  developed  classifications  of  analytical  activities 
which  have  emerged,  namely:  operations  analysis,  operations  research,  systems 
research,  systems  engineering,  cost- effectiveness  and  management  science. 

It  is  most  difficult  to  determine  what  the  exact*dcfinition  of  each  is  and 
which  one  of  the  subject  titles  subsumes  the  others. 

'Taylor,  F,  W„  Scientific  Management;  Harper  &  Bros.,  New  York,  1947 

^Trefethen,  F.  N.,  Operations  Research  for  Management;  The  John  Hopkins  Press, 
Baltimore,  Md.,  1954 

°Flagle,  C.,  et.al.,  Operations  Research  &  Systems  Engineering;  The  John  Hopkins 
*Press,  Baltimore,  Md.,  I960,  p.19. 

In  order  to  explore  areas  of  difference  in  understanding  about  Operations 
Research  and  Systems  Engineering  activities,  it  would  be  well  for  the  reader 
to  refer  to  the  following  books  and  periodicals: 

Bronowski,  J.;  Scientific  American.  Vol.  185,  October  1951,  pp. 75-77. 

Machol,  R.E.;  Mechanical  Engineering.  Vol.  79,  No.  9,  September  1957, 
pp. 890-91. 

Flagle,  C.,  et.al.  ;  Operations  Research  &  Systems  Engineering;  The  John 

Hopldns  Press,  Baltimore,  Md.,  1960,  p.19. 

Hall,  A.  D.t  A  Methodology  for  Systems  Engineering:  D.  Van  Nostrand  Co., 

Inc.,  Princeton,  New  Jersey,  1962. 
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After  World  War  11,  the  RAND  Corporation  interpreted  weapons  systems  analy¬ 
sis  as  a  description  of  those  studies  which  did  not  have  clearly  defined  inputs 
for  given  objectives  and  whose  future  uncertainties  were  recognized  t  o  be  less 
well  defined  than  those  of  earlier  studies. 

The  post  war  studies  in  weapons  systems  analysis  by  RAND  and  other  companies 
is  the  genesis  of  the  term  Systems  Analysis.  Charles  Hitch,  formerly  of  RAND, 
became  Assistant  Secretary  of  Defense,  Comptroller  in  1961  and  introduced  the 
concept  of  S-s  stems  Analysis  within  the  Department  of  Defense. 

Since  1961,  the  term  Systems  Analysis  has  been  used  by  DoD  to  describe  both 
the  philosophy  and  some  of  the  techniques  and  methodology  applicable  to  defense 
programming  and  budgeting  . 

In  Analysis  for  Military  Decisions,  E.  S.  Q-uade  describes  System  Analysis, 

"While  it  does  make  use  of  much  of  the  same  mathematics  (as  operations 
research)  -  it  is  associated  with  that  class  of  problems  where  the  diffi¬ 
culty  lies  in  deciding  what  ought  to  be  done  -  not  simply  how  to  do  it. 

The  total  analysis  is  thus  likely  to  be  a  more  complex  and  less  neat 
and  tidy  procedure,  one  seldom  suitable  for  quantitative  optimization. 

In  fact,  the  process  is  to  a  large  extent  synthesis:  the  environment  will 
have  to  be  forecast,  the  alternatives  designed  and  the  operational  laws 
invented.  Thus  with  a  systems  analysis,  one  associates  "broad"3  "long 
range",  "high  level",  "choice-of-objectives" ,  problems  and  "choice  of 
strategy",  "  qualitative  judgment"  and  "Assistance  to  logical  thinking". ^ 

In  a  later  definition,  E.  S.  Quade  states: 

"System  Analysis  -  a  systematic  approach  to  helping  a  decision-maker 
choose  a  course  of  action  by  investigating  his  full  problem,  searching 
out  objectives  and  alternatives  and  comparing  them  in  light  of  their 
consequences,  using  an  appropriate  framework  -  insofar  as  possible, 
analytic  -  to  bring  expert  judgment  and  intuition  to  bear  on  the  prob¬ 
lem.  "8 

This  latter  more  explicit  view  of  System  Analysis  seems  to  be  necessary 
in  view  of  the  increasing  sophistication  of  technical  programs  and  studies 
which  continually  cause  the  decision-maker (s)  to  need  more  capacity  for  under¬ 
standing  and  recommending  the  "best  approach".  The  nature  of  systems  analysis 
and  its  objectives  are  aimed  towards  this  goal. 

Most  of  .the  material  presented  above  relates  to  the  history  and  interpre¬ 
tation  of  systems  analysis  as  viewed  by  DoD. 

'Quade,  E.  S.,  Analysis  for  Military  Decisions;  Defense  Documentation  Center, 
Alexandria,  Va.,  AD  453887,  November  1964,  p.7. 

'Originally  appeared  in  the  book  SYSTEMS  ANALYSIS  AND  POLICY  PLANNING,  Applica¬ 
tions  in’ Defense,  Edited  by  E.  S.  Quade  and  W.  I.  Boucher;  Published  in  1968 
by  American  Elsevier  Publishing  Company,  Inc. 
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Nonmilitary  use  of  Systems  Analysis  has,  today,  culminated  in  development 
of  management  information  systems.  These  MIS  are  the  final  output  of  the  efforts 
of  Systems  Analysis  with  the  same  stipulated  objective  as  DoD  programming  - 
that  of  providing  the  maximum  cogent  information  to  a  decision-maker  for  a  given 
purpose. 

Nonmilitary  organizations,  generally,  do  not  have  the  inherent  complexity 
of  determining  the  optimum  solution  to  a  national  defense  posture  for  a  given 
time  period,  but  do  have  relatively  high  order  of  complex  problems  in  such 
areas  as  space,  management  science,  planning  and  forecasting,  resources  manage¬ 
ment,  product  line  mixes,  transportation,  communications  and  participation  in 
social  welfare  programs. 

As  can  readily  be  seen,  the  problems  are  somewhat  similar  in  total  objec¬ 
tive  --  The  best  decision.  However,  some  of  the  factors  aiding  utility  in 
civilian  Systems  Analysis  are:  costs  are  more  readily  determinable;  competition 
aspects  are  more  quantified  and  the  technology  is  at  hand  (or  can  be  determined 
within  closer  limits  than  can  that  of  the  military). 

The  stated  aim  of  materiel  Systems  Analysis  is  to  insure  that  the  Army 
can  accomplish  its  mission  within  the  level  of  effectiveness  specified  and  with 
the  minimum  expenditure  of  resources. 

This  goal  encompasses  resource  management;  and  although  costs  have  been 
implied  in  the  foregoing  discussion,  it  now  becomes  necessary  to  determine  how 
they  were  derived  and  how  they  interface  with  Systems  Analysis. 

1.3.2  History  of  Cost-Effectiveness 

Throughout  history,  man  has  reckoned  with  the  cost  of  the  item  he  acquired. 
Somehow,  through  mutual  agreement,  or  other  philosophy,  man  decided  what  the 
payment  (cost)  should  be  for  what  he  received. 

Early  Greek  philosophy  gave  us  the  word  Economics  -  then  defined  as  house¬ 
hold  management  -  which,  today,  is  designated  as  the  branch  of  social  science 
dealing  with  the  description  and  analysis  of  the  production,  distribution  and 
consumption  of  goods  and  services. 

Economic  philosophy  started  with  the  "philosophists  school^ .  Aristotle 
and  St.  Thomas  Acquinas  (comprehensive  codification  of  "just-price"),  then 
to  the  modem  age  economists  and  the  "classical  school"  (Adam  Smith,  John 
Stuart  Mill,  et.al.),  then  the  "Utopian  Socialists"  (Robert  Owen)  and  "Scien¬ 
tific  Socialists"  (Karl  Marx). 
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Today  most  economic  theory  is  classified  as  being  neo-classical  synthesis, 
i.e.,  a  marriage  between  micro  and  macroeconomics. 

The  essence  of  neo-classical  synthesis  is  the  modem  day  interpretation 
of  economic  analysis.  In  terms  of  J.  M.  Keynes:  9 

"The  object  of  our  analysis  is  not  to  provide  a  machine  or  method  of 
blind  manipulation  which  will  furnish  an  infallible  answer,  but  to  pro¬ 
vide  ourselves  with  an  organized  and  orderly  method  of  thinking  out  our 
particular  problems;  and,  after  we  have  reached  a  provisional  conclusion 
by  isolating  the  complicating  factors  one  by  one,  \\e  then  have  to  go 
back  on  ourselves  and  allow,  as  well  as  we  can,  for  the  probable  inter¬ 
actions  of  the  factors  amongst  themselves.  This  is  the  nature  of 
economic  thinking." 

Interpretation  of  the  above-mentioned  economic  philosophy  certainly  shows 
the  genesis  of  modem  analytical  thought  that  is  now  embodied  in  the  definition 
of  Systems  Analysis  and/or  Cost-Effectiveness. 

When  this  philosophy  is  combined  with  the  theory  of  Production;  Theory  of 
Input-Output  Analysis  (see  also  linear  programming);  Economic  Welfare  Theory; 
and  such  economics-oriented  definitions  as  Cost  (Resources);  Goods  and  Services; 
Value,  Price  and  Utility,  (see  also  marginal  utility),  it  becomes  apparent  that 
within  the  concept  of  neo-classical  synthesis  lies  the  springboard  from  our 
definition  of  Cost-Effectiveness. 

More  specifically.  Economic  Welfare  Theory  constituents  of  positive 
theory  and  welfare  theory  describe  the  evolution  as  such: 

Positive  theory  considers  the  development  of  economic  principles  of 
operation  regardless  of  desirability  or  not. 

Welfare  theory  is  concerned  with  the  evaluation  of  the  operation  of 
the  economy  in  terms  of  assumed  standards. 

The  overall  objective  of  Welfare  Economics  is  stated  in  the  term  Benefit-Cost 
Analysis:  A  means  of  estimating  the  prospective  economic  returns  of  a  pro¬ 
ject  (or  projects)  in  relation  to  costs. 

Comparison  of  Benefit-Cost  Analysis  and  Cost-Effectiveness  leaves  little 
doubt  as  to  the  specific  genesis  of  the  term  Cost-Effectiveness  or  Cost- 
Effectiveness  Analysis  as  defined  herein. 

The  evolution  of  the  term  "Cost-Effectiveness  Analysis"  occurred  after 
World  War  II  (see  paragraph  1.3.1  under  Systems  Analysis).  Cost-Effectiveness, 
per  se,  appears  to  have  been  formally  introduced  during  the  period  from  1961 


'Keynes,  J.  M.:  General  Theory  of  Employment,  Interest  and  Money;  Harcourt, 
Brace;  Nbw  York,  1936,  p.297. 
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to  1964.  This  is  evidenced  by  inclusion  of  cost-effectiveness  requirements 
formally  stipulated  in  certain  type  Request  for  Proposal  development/procure- 
ments  contracting  efforts  in  accordance  with  DoD  Directive,  (Series  3900.9; 

1964)  and  The  Contract  Formulation  -  Contract  Definition  Concept  Programming 
of  the  DoD  during  this  and  ensuing  periods  of  time. 

1.3.3  Systems  Analysis/Cost -Effectiveness 

The  literal  combining  of  the  terms  Systems  Analysis  and  Cost-Effectiveness, 
in  view  of  their  previously  developed  history  and  subsequent  definition  could, 
upon  examination,  raise  much  doubt  about  what  each  does  that  the  other  doesn't. 

Immediate  questions  are: 

Can  they  be  combined  as  SA/CE? , 

What  do  they  each  mean  in  this  form?. 

What  methodology  combines  them?,  and 
Aren't  they  interpretatively  redundant? 

Previous  history  and  definition  of  Systems  Analysis  illustrates  that  it 
is  more  likely  to  deal  with  that  class  of  problems  directed  towards  what  should 

be  done,  not  the  methodology  of  how  to  do  it.  In  this  sense,  then,  it  is  di¬ 

rected  at  the  suitability  of  implementing  a  specific  method  and  the  consideration 
of  alternatives  directed  towards  the  implementation  of  this  method. 

When  the  effort  is  directed  towards  the  costs  (and/or  resources)  required 
between  these  alternatives,  and  the  effect  of  changes  in  either  cost  or  effec¬ 
tiveness,  relative  to  each  other  and  mission  objectives,  then  \\c  use  the  term 
cost-effectiveness  analysis. 

The  objective  of  cost-effectiveness  is,  usually,  to  minimize  the  costs 
(resources)  at  which  a  given  level  of  effectiveness  can  be  attained  for  a  given 

mission.  This  also  includes  the  various  supporting  functions. 

In  order  to  further  clarify  the  specific  definition  of  each  term  and  to 
illustrate  their  integration,  the  reader  is  referred  to  the  immediate  following 
sections,  1.4  (Methodology  of  SA/CE)  and  1.5  (Application  of  SA/CE).  Also, 
the  subsequent  chapters  of  this  Guidebook  are  directed  towards  defining  the 
role  of  Systems  Analysis  and  Cost-Effectiveness  as  they  are  considered  in  the 
ensuing  analytical  formats. 

1.4  METHODOLOGY  OF  SYSTEMS  ANALYSIS/COST-EFFECTIVENESS 

There  isn't  any  singular  formulation  nor  is  there  a  standard  methodology 
which  is  applicable  "across-the-board"  that  allows  Systems  Analysis/Cost- 
Effectiveness  studies  to  be  performed  for  all  classes  or  sub-classes  of  problems 

explicitly . 
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The  immediate  lack  of  an  analytical  "cookbook"  approach  doesn't  preclude  the 
implementation  of  Systems  Analysis/Cost-Effectiveness  studies  however.  The  main 
virtue  of  any  stipulated  scientific  method  or  programming  function  is  its  recog¬ 
nition  of  change.  Comparison  of  a  generalized  "x-step"  scientific  approach  with 
the  Systems  Analysis/Cost-Effectiveness  methodology  presented  herein  reveals 
both  to  be  dynamic,  adaptive  processes.  The  singular  discrete  difference  is  in 
problem  formulation  and  solving  activities,  due  primarily  to  differences  in 
levels  of  abstraction. 

In  order  to  perform  any  analysis  it  is  necessary  to  conceive  a  disciplined 
framework,  i.e.,  a  systematic  approach,  with  provisions  for  making  comparisons 
between  alternative  ways  of  accomplishing  an  objective  systematically  (hopefully 
quantitatively)  in  a  logical  format  that  can  be  retraced  and  verified. 

Systems  Analysis  and  Cost-Effectiveness  studies  utilize  the  same  basic 
framework  for  their  objectives;  therefore,  it  now  becomes  necessary  to  differ¬ 
entiate  between  them  in  terms  of  the  definitions  given  in  sections  1.3.1  and 
1.3.2.  The  main  difference  appears  to  be  in  the  degree  of  applied  emphasis. 

When  the  study  is  directed  towards  the  determination  of  "costs"  between  similar 
systems  that  can  attain  a  specific  objective,  the  term  Cost-Effectiveness  analy¬ 
sis  is  applied.  When  the  problem  is  one  of  broader  scope;  i.e.,  consideration 
of  different  types  of  systems  that  could  attain  the  specific  objective,  then  the 
term  Systems  Analysis  is  used. 

Decisions  pertaining  to  choices  of  alternative  weapon  systems  or  force  struc¬ 
tures  and  the  strategies  for  their  employment  are  essentially  matters  of  economic 
choice.  Certain  elements  have  evolved  which  arc  common  to  these  kinds  of  deci¬ 
sions  and  have  been  contained  in  Systems  Analysis/Cost-Effectiveness  studies.  10 

1.  Objective  -  Systems  Analysis/Cost-Effectiveness  studies  are  initiated 
in  order  to  aid  in  determining  a  particular  policy  and/or  procedure.  These 
analyses  are  directed  toward  a  description  of  the  objectives  -  what  they 
should  be  (or  are).  This  done,  the  various  policies  and  procedures  are 
evaluated,  compared  and  "scaled"  in  order  to  determine  what  their 
effectiveness  and  costs  are  and  to  what  degree  they  do  "attain"  the 
objectlve(s) . 

2.  Alternatives  -  These  are  the  various  means  that  can  be  used  to  attain 
the  objective.  The  alternatives  presented  should  include  all  known  methods 
(also,  within  a  given  time  frame,  consideration  of  new  means  within  the 
"then"  known  state- of- the- art)  that  can  achieve  the  desired  results. 

The  alternatives  can  be  not  quite  obvious  and  consideration  of  all  types 
and  ways  of  doing  things  must  be  included.  (As  an  example,  if  the  ob¬ 
jective  of  any  given  period  of  history  was  peace  -  one  philosophy  was 
to  negotiate  -  another  was  war.  War  was  a  means  of  attaining  peace 
by  unification.  ) 

i0Quade,  E.  s.3  Analysis  for  Military  Decisions;  AD  453887,  p.155. 
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3.  Costs  -  The  costs  are,  also,  not  readily  obvious.  These  are  the  sun 
total  of  the  resources  expended  to  attain  the  objective(s)  for  each  of 
the  proposed  alternatives.  Resources  (costs)  are  those  items  consumed 
in  the  attainment  of  the  stipulated  objective  which  cannot  be  used  for 
other  purposes.  Total  costs  must  include  consideration  of  a  1 1  the 
factors  involved  in  accomplishing  an  objective.  (As  an  example,  the 
cost  of  smoking  is  determinable  at  "y"  cents  per  pack.  However,  if 

the  U.  S.  Surgeon  General's  Report  is  correct,  in  that  smoking  reduces 
life  by  (on  the  average)  8.3  years,  then  the  value  of  reduced  life 
is  a  cost  to  be  determined  and  applied  to  the  sum  total.) 

4.  Model  -  The  model  is  a  representation  of  the  reality  of  the  situa¬ 
tion  or  condition  being  studied.  Ideally,  it  would  represent  the  real 
situation  without  error  or  uncertainty .  Usually,  in  Systems  Analysis/ 
Cost  Effectiveness  studies  it  can  simulate  (at  best)  most,  or  some  por¬ 
tion,  of  the  real  world.  The  model  defines  its  representation  of  the 
real  world,  and  through  various  exercises,  simulations,  gaming  and 
mathematical  representations,  supplies  numerics  or  information  on  the 
effectiveness  of  the  various  alternatives  under  consideration  for  use 
in  attainment  of  objectives. 

The  structure  and  capability  of  the  model  is  a  major  limiting  fea¬ 
ture  of  an  analysis. 

A  basic  requirement  of  any  model  is  that  it  should  provide  correct 
answers  to  the  stated  questions  in  an  economical  manner.  This  causes 
consideration  of  the  following  factors:  representativeness,  uncertainty, 
data  sources  and  validity;  i.e.,  consistency,  sensitivity,  plausibility, 
criticality,  workability,  and  suitability. 

5.  Criteria  -  The  criteria  are  the  standards  or  accepted  rules  which 
are  used  to  determine  the  relativeness  or  desirability  of  one  alterna¬ 
tive  vs.  another  and  allows  for  choosing  one  in  preference  to  others. 

In  a  Cost-Effectiveness  analysis  it  provides  for  weighing  and  combining 
cost  vs.  effectiveness. 

At  this  point  in  Systems  Analysis/Cost-Effectiveness  studies  we 
can  now  interrogate  the  major  operating  elements  of  the  scientific 
methodology  within  the  general  framework  of  the  processes  of  a  system¬ 
atic  analysis.  Regardless  which  procedure  of  scientific  inquiry  is 
invoked,  the  analysis  proceeds  through  the  following  typical  stages: 

Hypothesis  -  At  this  step,  the  objective(s)  are  defined.  The 
constituent  elements  identified,  and  the  extent  of  the  problem 
delimited  to  suit  knowledge,  time  and  cost  considerations. 

Definition  -  This  step  explores  the  alternate  configurations, 
policies  or  programs  that  can  be  directed  towards  solution  of  the 
problem  (objective).  Inherent  in  this  step  are  considerations  of 
the  resources  and  other  interrelationships. 
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Analysis  -  Construction  of  the  model (s)  at  the  necessary  level  of 
abstraction;  and  exercising  the  model  to  determine  the  consequences 
of  alternate  programs  and  considered  factors. 

Evaluation  -  This  step  examines  the  derived  results.  It  is  here 
that  the  preferred  alternative  is  identified.  The  evaluation  is 
based  on  modification  of  all  factors  discovered  in  the  iterative 
analysis . 

Conclusion  -  This  step  is  concerned  with  the  verification  of  the 
resultant  analysis  by  test  and/or  experimentation. 

The  above  application  of  the  systematic  approach  is  not  new  and  is  quite 
straight  forward.  The  problems  encountered  in  using  the  approach  are  not  caused 
by  its  non-applicability,  but  because  of  the  vagaries  of  the  environment  we  are 
attempting  to  use  it  in  and  for. 

In  Systems  Analysis/Cost-Ef fectiveness  studies  the  decision-maker  and 
analysts  would  like  to  live  and  operate  in  a  deterministic  atmosphere,  and, 
at  worst,  in  a  well  behaved  stochastic  environment.  In  this  environment,  the 
decision-makers/analysts  can  modify  hypotheses,  which  are  sub j ectively  probabi¬ 
listic  at  worst,  with  information  gleaned  from  application  of  a  defined  scien¬ 
tific  methodology,  causing  revision  of  the  original  probabilistic  function 
towards  a  fully  descriptive  and  validated  function  which  expresses  the  actual 
environment  exactly. 

Unfortunately,  the  real  world  precludes  such  a  standardized  and  ideal 
approach  and  solution  to  problems  encountered  by  present  day  Systems  Analysis/ 
Cost-Effectiveness  studies. 

Consider  the  objective  of  military  posture  at  some  time  in  the  future.  What 
should  its  composition  be  and  what  should  it  be  capable  of?  At  once,  the  objec¬ 
tives  are  not  even  known,  are  certainly  multiple  and  what  can  be  postulated  as 
a  figure  of  merit?  Knowing  that  the  Systems  Analysis  function  would  be  to  con¬ 
sider  not  only  pure  military  posture,  but  with  consideration  of  the  interrela¬ 
tionships  between  socio-economic  and  political  factors  and  military  affairs, 
it  becomes  immediately  clear  that  a  "model"  would  be  quite  difficult  to  con¬ 
struct, 

The  analytical  framework  postulated  above  cannot  accomplish  analyses  such 
as  is  necessary  in  view  of  these  objectives  in  a  single  pass-through.  In  order 
to  maximize  the  amount  of  information  that  can  be  obtained  from  such  an  ap¬ 
proach,  it  is  necessary  to  iteratively  exercise  the  framework.  Selection  of 
objectives,  alternatives,  data  collection,  modelling  activities,  establishing 
measures  of  effectiveness  and  figures  of  merit,  sensitivity  analyses,  evalua¬ 
tions,  modifications  and  conclusions  should  be  iterated  through  the  established 
analytical  framework  in  an  attempt  to  remove  the  "impurities"  of  the  first 
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run-through  and  reduce  the  overall  complex  subjective  area  into  a  quantifiable 
real  world  area  as  near  as  can  be  accomplished. 

The  iterations  produce:  delimited,  redefined  or  changed  objectives  (per¬ 
haps  even  causing  suboptimization  to  be  considered  in  view  of  total  complexity); 
discovery  of  new  alternatives  as  by-products  of  analysis,  or  modification  of 
type  and  quantities;  redetermination  of  costs  or  resources  due  perhaps  to  con¬ 
sideration  of  having  to  consider  non-dollar  cost  contingencies;  necessity  of 
modelling  changes  in  order  to  more  accurately  depict  the  "real  world"  consider¬ 
ing,  also,  the  constraints  and  configurations  in  terms  of  new  data  or  redefini¬ 
tion  of  objectives  and  measures  of  effectiveness  and  establishment  of  new 
criteria  in  terms  of  new  data,  information  and/or  changes  in  the  inherent  ele¬ 
ments  in  the  analysis  or  in  more  accurate  standards  or  rules  so  delineated  by 
recursion. 

Obviously,  the  process  can  continue  indefinitely;  but  normally,  it  is 
exercised  until  the  results  are  deemed  satisfactory  or  the  constraints  of  time 
and/or  money  force  discontinuance. 

At  this  point  in  the  dessertation  it  is  necessary  to  digress  from  the  prime 
purpose  of  this  section  -  Methodology  of  Systems  Analysis/Cost-Effectiveness  - 
which  explains  the  how  it  is  done,  and  ask  win'  and  for  whom.  Subsequent  sec¬ 
tions  explain  what  and  when  as  well  as  modification  of  how  and  why;  but,  for 
whom  needs  a  brief  explanatory  section. 

This  entire  SA/CE  studies- analysis  is  directed  towards  improving  the  quan¬ 
tity,  quality,  and  accuracy  of  information  that  is  necessary  (or  would  be  help¬ 
ful)  for  a  decision-maker,  in  order  that  he  may  make  the  best  possible  decision 
with  minimum  uncertainty  and  risk  -  or,  if  you  will  -  the  decision  as  to  what 
is  more  effective,  economical  and  timely.  As  such,  SA/CE  is  a  prime  management 
tool,  nothing  more  -  hut  nothing  less. 

Granted,  the  genesis  of  the  reason  or  need  for  a  study  is  based  on  specu¬ 
lation  of  "decision-makers"  and  that  it  does  contain  subjective  value  judgments, 
lack  of  precise  knowledge  and/or  data,  uncertainty  of  competitive  strategy  and 
other  uncertainties.  However,  the  studies  can  aid  the  decision-makers  by 
assessment  of  implications  gleaned  from  selection  of  various  alternatives. 

Systems  Analysis  concerns  itself  with  problems  of  resource  allocation,  i.e., 
what  mix  of  "things"  should  be  obtained  and  how  long  are  they  to  be  considered 
adequate  for  their  purpose.  Cost-Effectiveness  reveals  the  cost  versus  effec¬ 
tiveness  of  the  "mix"  and  aids  in  determination  of  whaf  is  the  best  way  to 
"spend"  the  resources. 
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Without  programming  this  philosophy  through  a  systematic  analytical  format, 
there  is  no  better  way  of  determining  how  many  of  what  can  accomplish  a  mission  - 
i.e.j  -  it  is  not  sensible  to  intuitively  prescribe  a  posture  without  considering 
whatever  numerics  can  be  gained  from  a  "study"  and  what  their  information  content 
is.  It  i  s  not  the  purpose,  nor  can  it  be  accomplished  by  SA/CE,  to  cause  a 
decision-maker  to  agree  beyond  all  doubt  that  this  information  does  indeed  pre¬ 
sent  the  course  of  action  to  follow. 

The  prime  purpose  is  to  provide  as  much  quantified  information  -  including 
limiting  conditions  whether  truly  quantified  or  qualitative  -  as  is  possible 
in  order  to  sharpen  the  intuition  of  the  decision-makers  and  help  them  arrive 
at  the  best  solution  in  terms  of  their  observation,  intuition,  experience  and 
value  judgments. 

Discussion  of  limitations  of  SA/CE  will  be  presented  in  the  final  segment 
of  this  chapter. 

1.5  APPLICATION  OF  SYSTEMS  ANALYSIS/COST-EFFECTIVENESS 

1.5.1  General 

The  concepts  and  philosophy  of  Systems  Analysis/cost-Eff ectiveness  can  be 
applied  to  almost  any  system  at  any  time  during  the  life  cycle. 

The  Aimy  Materiel  Command  (AMC)  states  that  a  requirement  for  a  Systems 
Analysis/Cost-Eff ectiveness  study,  evaluation,  or  investigation  may  exist  or 
be  initiated  in  support  of  concept  formulation  (CF),  contract  definition  (CD), 
program  change  requests  (FCR's)*  program  submissions,  technical  feasibility 
studies  (FS's),  and  studies  associated  with  qualitative  material  development 
objectives  (QMDO's),  qualitative  material  approaches  (QMA's),  advanced  develop¬ 
ment  objectives  (ADO's),  advanced  development  plans  (ADP's)  and  technical 
development  plans  (TDP’s).  In  addition,  other  phases  in  the  life  cycle  where 
this  type  of  evaluation  should  be  applied  are  in  long  range  R  &  D  systems  plan¬ 
ning,  maintainability  and  reliability  studies,  as  well  as  in  major  inventory 
and  logistics  decisions. 

The  study  may  take  on  any  of  the  forms  in  Figure  1-1,  depending  upon  where 
it;  is  in  the  life  cycle  as  well  as  what  type  of  decision  must  be  made. 


-* - 

AMCR  11-1,  21  April  1970. 
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(1)  System-Effectiveness  Studies 

•  Investigates  technical  feasibility 

.  Predicts  how  well  a  system  will  meet  mission  requirements 


(2)  System- Analysis  Studies 

•  Technical- feasibility  studies  and  concept  selection 


•  Effectiveness  Analysis 

•  Cost  Analysis 


Inter-System  trade-offs 


(3)  Cost-Effectiveness  Studies 

•  Effectiveness  Analysis 

•  Cost  Analysis 


Intra-Systan  trade-offs 


FIGURE  1-1 

TYPES  OF  SYSTEMS-ANALYSIS  STUDIES 


Category 

Orientation 

SA/CE  Milestones 

Research  (6.11) 

Increase  knowledge 

Exploratory  Development  (6.21) 

Technical  feasibility 

Technical  feasibility  and 
concept  selection  studies 

Advanced  Development  (6.31) 

Operational  and  tech¬ 
nical  suitability 

Inter-system  C-E  studies 

Engineering  Development  (6.41) 

Design  Engineering 

Intra- system  PD/CE  studies 

Operational  Systems  Develop¬ 
ment  (6.71) 

Production  Engineering 

FIGURE  1-2 
RDT&E  CYCLE 
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The  relationship  of  the  types  of  studies  to  the  major  phases  in  the  RDT  &  E 
Cycle  is  contained  in  Figure  1-2.  During  exploratory  development  (6.21)  the 
type  of  system  analysis  studies  that  are  conducted  are  normally  concerned  with 
technical  feasibility  and  concept  selection.  However,  when  advanced  develop¬ 
ment  (6.31)  begins,  the  primary  aim  is  to  conduct  inter-system  trade-offs  in 
order  to  choose  among  several  alternative  systems  capable  of  performing  some 
given  mission,  assuming  that  all  contending  systems  are  capable  of  performance 
at  various  levels  of  effectiveness. 

In  engineering  development  (6.41)  parametric  design/cost  effectiveness 
(PD/CE)  studies  are  conducted.  This  is  a  process  of  formulating  and  evaluating 
a  complete  range  of  alternative  intrasystem  trade-offs  of  components  (i.e,, 
designs)  to  provide  the  optimum  capability  for  fulfilling  a  given  system  mission. 

1.5.2  Application  of  SA/OE  to  Concept  Formulation  and  Contract  Definition 

Recent  DoD  and  Army  Directives*  which  establish  the  concept  formulation  and 
contract  definition  phases  of  the  system  life  cycle  show  an  increasing  aware¬ 
ness  of  and  need  for  Army  program  managers  to  make  sound  decisions  based  upon 
quantitative  evaluations  which  should  result  in  economical  and  operationally 
effective  system  designs  capable  of  meeting  the  desired  performance  require¬ 
ments.  In  this  section,  concept  formulation  and  contract  definition  will  be 
described  and  the  requirements  for  Systems  Analysis/Cost-Ef f ectiveness  studies 
will  be  discussed. 

The  objective  of  concept  formulation  is  to  provide  the  technical,  economic, 
and  military  basis  for  a  conditional  decision  to  initiate  engineering  develop¬ 
ment  . 

It  is  accomplished  through  comprehensive  system  studies  in  exploratory  and 
advanced  development  by  means  of  experimental  tests,  engineering  and  analytical 
studies.  This  work  constitutes  the  necessary  preliminary  threat  and  operational 
analyses,  trade-off  and  cost  effectiveness  studies,  and  development  of  components 
and  technology  -  to  assure  a  firm  foundation  for  Engineering  Development.  The 
evidence  required  for  a  conditional  decision  to  proceed  with  Engineering  Develop¬ 
ment  includes  the  following  prerequisites: 

(a)  Primarily  engineering  rather  than  experimental  effort  is 
required,  and  the  technology  needed  is  sufficiently  in  hand. 

(b)  The  mission  and  performance  envelopes  are  defined. 

(c)  The  best  technical  approaches  have  been  selected. 

(d)  A  thorough  trade-off  analysis  has  been  made. 

*DoD  Directive  3200.9,  AR  705-5- 
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(e)  The  cost-effectiveness  of  the  proposed  item  is  favorable 
in  relation  to  that  of  competing  items  on  a  defense-wide 
basis. 

(f)  Cost  and  schedule  estimates  are  credible  and  acceptable. 

Qi  the  basis  of  this  information,  the  Anny  requests  approval  to  initiate 
Engineering  Development.  The  request  is  made  either  by  memorandum  to  DDR  &  E, 
or  if  required,  by  a  Program  Change  Request  (PCR).  It  is  accompanied  by  a 
Technical  Development  Plan  (TDP),  specifically  addressed  to  the  six  prerequi¬ 
sites  cited  above,  which  summarizes  pertinent  cost-effectiveness  studies  and 
developments  and  provides  whatever  information  may  be  required  to  substantiate 
the  achievement  of  these  prerequisites. 

If  the  initiation  of  Engineering  Development  receives  conditional  approval, 
the  Contract  Definition  phase  begins.  The  objective  of  Contract  Definition  is 
to  determine  whether  the  conditional  decision  to  proceed  with  Engineering 
Development  should  be  ratified.  Its  ultimate  goal  is  achievable,  firm  and 
realistic  performance  specifications,  backed  by  a  firm  fixed  price  or  fully 
structured  incentive  proposal  for  Engineering  Development.  In  addition,  it 
embraces  the  following  subsidiary  objectives: 

(a)  Precisely  define  interfaces  and  responsibilities. 

(b)  Identify  high  risk  areas. 

(c)  Verify  technical  approaches. 

(d)  Establish  firm,  realistic  schedules  and  cost  estimates  for 
Engineering  Development,  including  production  engineering, 
facilities,  construction,  and  production  hardware  that  will 
be  funded  during  Engineering  Development  because  of  con¬ 
currency  consideration. 

(e)  Establish  schedules  and  cost  estimates  for  planning  purposes 
for  the  total  project,  including  production,  operation,  and 
maintenance. 

The  Contract  Definition  procedure  is  mandatory  for  all  new  Engineering 
Developments  or  Operational  Systems  Developments  (or  major  modifications  of 
existing  ones)  that  are  estimated  to  require  total  cumulative  research,  develop¬ 
ment,  test,  and  evaluation  financing  in  excess  of  $25  million,  or  a  total  pro¬ 
duction  investment  in  excess  of  $100  million.  (However,  DoD,  DA,  or  AMC  may 
require  Contract  Definition  on  other  systems  which  are  below  the  $25  million 
and  $100  million  threshholds.) 

Contract  Definition  is  normally  performed  by  two  or  more  contractors  in 
competition  under  the  technical  direction  of  the  cognizant  Army  activity. 

It  may,  however,  be  performed  by  a  sole  source  contractor  if  necessary. 
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The  trade-off  studies  that  are  conducted  during  this  phase  should  be  directed 
toward  achieving  an  optimum  balance  between  total  cost,  schedule,  and  opera¬ 
tional  effectiveness  for  the  system.  Total  cost  (or  life  cycle  cost)  includes 
the  cost  of  development,  production,  deployment,  operation,  and  maintenance. 
Operational  effectiveness  includes  all  the  factors  that  influence  effectiveness 
in  operational  use,  as  well  as  inherent  or  pure  performance  characteristics 
(as  in  WSEIAC.  the  ADC  matrices). 

The  system  includes  not  only  the  hardware  but  also  all  other  required  items, 
such  as  facilities,  data,  training  equipment,  and  the  operational  and  support 
personnel  who  will  be  required. 

The  end  product  of  Contract  Definition  is  a  complete  technical,  manage¬ 
ment,  and  cost  proposal  package  for  Engineering  Development.  The  contractor's 
package  should  include  such  information  as  a  List  of  the  end  items  required; 
performance  specifications  for  each  item;  a  work  breakdown  structure  and  a 
PERT  network  plan;  the  principal  objectives  and  features  of  the  overall  system 
design,  including  recommendations  for  its  operational  use;  a  recommended  main¬ 
tenance  plan;  detailed  cost  estimates  and  milestone  schedules  for  five  years 
beyond  it;  quantitative  reliability  and  maintainability  specifications  and 
test  plans;  time/cost  performance  trade-off  decisions  on  major  alternatives; 
required  new  designs  and  technology;  foreseeable  technical  problems  and  pro¬ 
posed  solutions;  technical  specifications  and  performance  specifications  for 
support  items  (facilities,  training  devices,  and  so  on)  for  which  early  Engineer¬ 
ing  Development  is  required;  delivery  schedules  and  requirements  for  data  and 
documentation;  and  a  proposed  schedule  of  production  engineering  and  production 
tooling  in  relation  to  Engineering  Development,  if  appropriate. 

After  a  review  of  the  contractors'  Contract  Definition  proposal  packages, 
the  Army  recommends  one  of  the  following  actions:  to  contract  for  Engineering 
Development  on  the  basis  of  the  proposals  received;  to  contract  with  an  alter¬ 
native  source;  to  continue  further  Contract  Definition  effort;  to  defer  or 
abandon  the  Engineering  Development  effort;  or  to  undertake  further  Exploratory 
or  Advanced  Development  of  key  components  and/or  systems  studies. 

1.5-3  Conclusion 

The  methodology  and  application  of  Systems  Analysis/Cos t-Eff ectiveness 
studies  is  directed  toward  supplying  the  decision-maker  with  the  maximum  mount 
of  quantifiable  information  about  alternative  approaches  to  attaining  a  mission. 
Also,  it  stipulates  areas  of  qualified  considerations  with  this,  and  allows  the 
virtues  of  a  systematic  approach  -  the  design  and  development  of  proposed  objec¬ 
tives  and  their  solutions  within  a  rigorous,  logical,  adaptive,  dynamic  framework 
which  can  be  retraced  and  verified. 
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1.6  LIMITATIONS  OF  SYSTEMS  ANALYSIS/COST-EFFECTIVENESS 

There  are  many  advantages  to  Systems  Analysis  and  Cost-Effectiveness.  How¬ 
ever,  they  are  not  panaceas  that  handle  all  problems  of  the  system  developer, 
manager,  or  user,  nor  are  they  without  limitations.  Systems  Analysis/Cost- 
Ef fectiveness  studies  must  be  examined  to  recognize  the  limitations  built  into 
them,  or  the  premises  generated  based  on  given  information. 

The  more  prominent  limitations  inherent  in  all  but  extremely  simple  analyti¬ 
cal  studies  are  as  follows: 

Inadequate  Problem  Definition 
Improperly  Defined  Scope 
Restriction  of  Alternatives 
Improper  Criteria 
Interjection  of  Bias 
Improper  Data  Usage 
Incompatible  Model 
'  Misapplication  of  Model 

Forcing  Problem  into  Improper  Framework 
Improper  Handling  of  Relevant  Factors 
Poor  Assumptions 
Ignoring  Uncertainties 
Misinterpreting  Model  Results 
Insufficient  Samples 
"  Failure  to  Reappraise 
Failure  to  Communicate 
Measures  of  Effectiveness  Approximate 
Future  Uncertainties 
Analysis  Never  Truly  Complete 
Changing  Value  Systems 
Neglect  of  Subjective  Elements 

Assignment  of  Value  to  "Costs"  (Economic  Costs) 

Inability  to  Verify  Decision 
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Consideration  of  each  element  described  as  a  limitation  is  not  the  purpose 
of  this  section.  However,  in  order  to  show  relevancy  of  limitations  and  the 
dangers  associated  with  their  non-consideration,  a  few  examples  will  be  given. 

First,  consider  the  list  of  limiting  elements  described  above.  If  an 
analysis  can  never  be  complete,  due  to  the  state  of  knowledge,  time  and  money, 
it  is  fair  to  assume  that  this  list  of  elements  is  not  complete. 

Consider,  also,  the  statement,  "Mustang  makes  it  happen"  as  an  aid  to  de¬ 
cision  making  about  buying  a  car.  If  I  an  a  car  buyer,  I  must  make  a  decision 
based  on  what  the  statement  means  to  me. 

I  buy  a  Mustang  and  wait  for  it  to  happen  and,  unless 
I  am  purely  adventurous,  I  assume  it  is  good. 

(b)  I  don1 1  buy  a  Mustang  because,  if  it  happens,  I  am 

in  no  position  to  deal  with  it;  or  it  is  bad  (in  my 
opinion)  and  I  don't  want  it  to  happen. 

Now.  either  of  these  decisions  can  be  a  misinterpretation  depending  on  the 
"true"  significance  of  it. 

The  real  difficulty  is  in  the  word  (information)  it;  what  problem  did  it 
answer?  The  same  is  true  with  results  of  analyses.  Do  they  literally  answer 
the  problem  or  do  I  still  have  to  interpret  meaning  within  ny  bias? 

As  another  example;  interpretation  of  a  real  weather  report  which  states  - 
"Probability  of  rain  -  90$  in  25$  of  the  area  10$  of  the  time".  I  would  assume 

that  there  is  a  good  chance  of  rain;  that  25$  of  the  area  will  get  rain;  and 

of  the  duration  of  time  (which  this  event  encompasses)  it  will  rain  10$  of  that 
period  of  time. 

Nxv  if  the  problem  is  where  do  I  go  so  that  I  won't  get  wet,  how  do  I 
interpret  this?  It  would  appear  I  would,  generally,  be  wrong  in  whatever  I 
decide,  or  have  minimum  confidence  that  try  decision  is  correct. 

This  example,  also  considers  dangers  from  the  elements  of  limitation  of 
poor  assumptions,  improperly  defined  "model"  and  failure  to  communicate,  at 
least.  Analogous  to  this  problem  is  the  one  of  forecasting  advanced  weaponry 
for  use  in,  say,  1985-  What,  how  many,  win'  and  against  whom?  Basically,  the 
hard  information  comes  from  back-feed  and  is  tempered  by  intuition  and  judgment 
to  transcend  now  to  the  future.  However,  until  the  "systems"  are  operational, 
in  their  then  environment  and  we  can  obtain  feedback,  we  will  always  have  error 
terms  in  the  analysis  -  until  we  pass  through  the  verification/modification 
stages. 
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Consider,  also,  data  errors  and  modelling  problems  in  determining  how  to 
attach  significance  to  deterrence.  Actually,  deterrence  is,  mostly,  a  matter 
of  philosophy.  What  does  an  enemy  consider  it  to  be  -  and  how  do  you  find 
out?  Also,  it  w  i  1 1  be  a  changing  value  with  time. 

Quantification  of  numerics  is  a  problem  in  the  area  of  "costs".  Hw 
accurately  can  you  predict  costs  for  the  future  -  even  in  dollars  -  much  less 
recognize  a  variable  depletion  of  resources  -  with  a  changing  value  system? 

Obviously,  it  is  impossible  to  consider  all  the  limitations  applicable  to 
SA/CE  analytical  functions,  as  some  apply  and  some  do  not,  depending  on  the 
complexity  of  the  problems. 

Yet,  the  recognition  of  them  and  the  degree  of  their  accountability  is  a 
by-product  of  an  analysis,  and  does  supply  information  as  much  as  the  hard 
numerics.  By  the  recognition  (and  measure)  of  accountability  attained,  the 
analytical  processes  become  better  by  degree  and  help  to  sharpen  the  intuition 
and  judgment  functions  of  the  decision-maker. 
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CHAPTER  2 

GENERAL  METHODOLOGICAL  APPROACH  TO  SYSTEMS  ANALYSIS/ COST- 

EFFECTIVENESS  STUDIES 


The  general  methodological  approach  to  the  systems  analysis/cost-effective¬ 
ness  process  is  shown  in  Figure  2-1. 

The  approach  shown  is  applicable  to  both  systems  analysis  and  cost-effec¬ 
tiveness,  The  distinction  between  systems  analysis  and  cost  effectiveness  is 
mainly  a  difference  in  the  definition  of  the  scope  of  the  study.  Systems 
analysis  studies  are  concerned  with  problems  of  large  scope  and  are  character¬ 
ized  by  the  comparison  of  different  types  of  equipments  or  systems  to  determine 
the  best  approach  for  meeting  some  stated  requirement.  Technical  feasibility, 
inter-system  trade-offs,  and  the  parameterizing  of  requirements  are  all  as¬ 
sociated  with  systems  analysis.  Cost-effectiveness  studies,  on  the  other  hand, 
deal  with  narrower  problems .  Normally,  in  cost-effectiveness  studies,  the  type 
of  system  to  be  analyzed  will  be  given,  and  the  problem  is  to  conduct  intra¬ 
system  trade-offs  to  "optimize"  the  system,  i.e.,  to  develop  the  best  system 
with  respect  to  performance,  cost,  schedule,  manpower,  etc. 

2.1  INPUT  INFORMATION 

The  initial  inputs  that  lead  to  the  development  of  requirements  and 
objectives  in  the  systems  analysis/cost-effectiveness  process  are  normally  at¬ 
tributable  to  such  documents  as  the  Army  Force  Development  Plan,  the  Combat 
Development  Objective  Guide,  a  Qualitative  Materiel  Development  Objective,  or 
a  Qualitative  Materiel  Requirement. 

2.1.1  Army  Force  Development  Plan  (AFDP) 

The  AFDP  is  a  responsibility  of  the  Assistant  Chief  of  Staff  for  Force 
Development  (ACSFOR)  and  is  constrained  by  anticipated  resource  limitations. 

It  provides  the  planning  basis  for  the  Five-Year  Force  Structure  and  Financial 
Program,  and  its  objective  is  to  provide  the  best  possible  Army  posture  within 
available  resources.  Specifically,  it  accomplishes  the  following: 

It  plans  the  development  of  balanced  capabilities  within  established 
constraints  and  strives  to  achieve  the  best  possible  balance  between 
forces,  readiness,  and  modernization. 
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•  It  plans  incremental  increases  in  capabilities  in  order  of  priority  and 
identifies  the  associated  additional  resources  necessary  to  attain  them 
in  a  reasonable  time. 

•  It  plans  incremental  decreases  in  capabilities  in  inverse  order  of 
criticality  to  provide  resources  to  meet  unprogrammed  requirements. 

2.1.2  Combat  Development  Objectives  Guide  (CDOG) 

The  CDOG,  prepared  by  the  USACDC,  provides  guidance  for  Army  combat- 
development  activities  and  the  research  and  development  program.  It  contains 
all  the  DA-approved  operational  and  organizational  objectives.  Qualitative 
Materiel  Development  Objectives,  Qualitative  Materiel  Requirements,  and  Small 
Development  Requirements  and  their  priorities.  It  also  contains  a  compilation 
of  studies,  field  experiments,  and  tests.  These  contents  are  defined  as 
follows: 

•  Operational  objective.  At  operational  objective  is  an  Army-approved  need 
for  a  new  or  improved  operational  capability  that  pertains  to  operational 
concepts,  tactics,  techniques,  and  procedures. 

•  Organizational  objective.  An  organizational  objective  is  an  Army-approv¬ 
ed  need  for  a  new  or  revised  organization  to  improve  Army  operational 
capabilities. 

•  Qualitative  Materiel  Development  Objective  (QMDO).  A  QMDO  is  an  Army- 
approved  statement  of  a  military  need  for  the  development  of  new  materiel, 
the  feasibility  of  which  cannot  be  determined  sufficiently  to  permit  the 
establishment  of  a  Qualitative  Materiel  Requirement  (WR)  . 

•  Qualitative  Materiel  Requirement  (QMR) .  a  QMR  is  an  Army- approved  state¬ 
ment  of  military  need  for  a  new  item,  system,  or  assemblage,  the  develop¬ 
ment  of  which  is  believed  feasible. 

•  Small  Development  Requirement  (SDR).  Ai  SDR  states  an  Army  need  for  the 
development  of  equipment  that  can  be  developed  in  a  relatively  short 
time  and  does  not  warrant  the  major  effort  required  in  developing  a  QMR. 

Ai  SDR  is  normally  considered  the  appropriate  requirements  document  if: 
(1)  development  of  the  item  is  of  proven  feasibility;  (2)  the  time  re¬ 
quired  for  development  is  2  years  or  less;  and  (3)  the  RDIE  costs  will 
not  exceed  $2.5  million,  and  the  projected  investment  of  IEM\  funds  will 
not  exceed  $10  million. 

2.1.3  Qualitative  Materiel  Development  Objective  (QMDO)  Approval 

The  first  formal  requirements  document  that  the  Army  uses  in  the  research 
and  development  cycle  is  the  QMDO.  Any  individual  (military  or  civilian),  unit, 
or  agency  may  propose  a  concept  or  idea  that  might  lead  to  establishing  a  QMDO. 
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However.  CDC  is  delegated  the  authority  and  responsibility  for  preparation  and 
submission  of  QMDO's  to  ACSTOR  for  approval.  CDC  uses  operations-research 
techniques  to  relate  future  tactics  to  the  technical  estimates  and  concepts 
furnished  by  AMC  as  a  result  of  its  basic  and  applied  research  work.  Evaluation 
of  requirements  for  materiel  for  the  Amy  dictates  continuous  liaison  between 
CDC  and  the  developing  agencies  of  CDC  at  the  laboratory  and  commodity  command 
level.  From  this  point  on  a  1 1  research  and  development  must  be  supported  by 
either  an  objectives  or  a  requirements  document  such  as  a  QMDO,  QMR,  or  SCR 
unless  otherwise  directed  by  appropriate  DA  authority. 

The  QMDO  normally  begins  with  requirements  generated  by  combat-development 
studies  conducted  by  CDC.  These  requirements,  when  reviewed,  indicate  whether 
they  are  within  the  state-of-the-art  or  have  been  sufficiently  exploited  to  per¬ 
mit  immediate  development.  The  QMDO  is  associated  with  requirements  necessitat¬ 
ing  further  research.  At  this  point  we  are  still  several  years  (3-8)  away  from 
establishing  firm  military  characteristics.  The  QMDO  states  in  general  terms 
the  objective,  the  operational  and  organizational  concepts  for  employment,  a 
description  of  full  justification  for  the  new  item,  and  its  priority  (I,  11,  or 
III),  Usually,  the  QMDO  is  first  circulated  in  draft  form  to  AMC  and  other 
interested  commands  for  comments  and  suggested  revisions.  AMC's  input  generally 
consists  of: 

•  A  "ball  park"  cost  figure  for  the  required  research  and  exploratory 
development 

■  A  time-frame  for  completion  of  the  work 

•  The  effect  performance  might  have  on  already  approved  programs 

•  The  potential  of  present  knowledge  to  advance  the  state-of-the-art  suf¬ 
ficiently  to  make  the  project  feasible 

QMDO's  may  be  initiated  at  any  stage  in  the  research- and- development 
cycle  and  may  result  from  a  combat- development  study,  operational  experience, 
developmental  experience,  technological  breakthroughs,  or  feedback  of  defi¬ 
ciencies  in  existing  equipment. 

The  Director  of  Research,  Development,  and  Engineering.  Headquarters, 
AMC,  is  the  AMC  focal  point  for  processing  all  new  materiel  objectives  docu¬ 
ments  and  for  ensuring  that  adequate  QMDO  plans  are  prepared.  After  review¬ 
ing  all  the  comments  from  other  agencies  on  the  draft  QMDO.  CDC  revises  it 
and  submits  it  to  DA  for  formal  staffing  and  approval.  ACSFOR  has  DA  staff 
responsibility  for  all  QMDO's.  After  DA  approval.  QMDO's  are  published  in 
CDOG  so  that  all  interested  commands  and  agencies  can  be  made  aware  of  the 
objectives. 
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2.1.4  Qualitative  Materiel  Requirement  (QMR) 

After  research  and  exploratory  development  have  progressed  to  the  point 
where  AMC  feels  it  may  be  possible  to  translate  technical  knowledge  acquired 
by  the  laboratories  into  a  feasible  system,  it  will  prepare  a  draft  QVR  or 
recommend  to  CDC  that  a  QMR  be  prepared  and  issued.  The  QMR  is  directed  toward 
attaining  new  or  substantially  improved  materiel  that  will  significantly  advance 
the  Army  ability  to  accomplish  its  mission.  Unlike  the  QMDO,  the  QMR  is  much 
more  specific  in  describing  the  requirement.  It  states  major  materiel  needs  in 
terms  of  military  characteristics  and  priorities  and  relates  materiel  to  the 
operational  and  organizational  context  in  which  it  will  be  used. 

2.2  DEFINE  REQUIREMENTS  AND  OBJECTIVES 

The  basic  task  of  systems  analysis  is  to  evaluate  systems  in  terms  of 
achieving  objectives  and  consuming  resources.  While  this  evaluation  might  be 
made  at  any  point  in  the  life  cycle  of  a  system,  major  interest  currently  lies 
in  making  such  evaluations  before  resources  are  committed  to  creating  a  system. 
Major  emphasis  in  this  section  is  therefore  given  to  such  prior  evaluations. 

In  any  systems  analysis,  it  is  essential  to  take  the  information  as  stated 
in  the  study  directive  and  define  an  acceptable  set  of  requirements  and 
objectives.  Stating  the  detailed  requirements  and  objectives  is  a  major  part 
of  the  study  effort;  they  may  require  updating  and  redefinition  following 
evaluation  and  feedback. 

In  establishing  the  objective,  it  is  necessary  to  define  the  boundaries 
of  the  analysis  in  terms  of  system  objectives,  system  definition,  associated 
operations,  and  influencing  factors.  Initially,  the  objective  should  be  stated 
in  general  terms  so  that  a  comprehensive  analysis  can  be  made.  Extreme  care 
and  effort  must  be  made  to  properly  state  the  system's  objectives.  It  is 
essential  to  the  entire  analysis,  since  it  is  the  major  factor  in  the  selection 
of  the  missions,  the  synthesis  of  alternative  systems,  the  evaluation  criteria 
as  well  as  dictating  the  types  of  models  which  must  be  employed  in  the  study. 

It  is  not  always  easy  to  decide  what  is  to  be  included  in  the  analysis.  Often 
the  decision  must  be  made  subjectively,  and  it  may  be  heavily  influenced  by 
such  factors  as  data  inadequacies,  schedule  and  manpower  limitations,  and 
uncertainties  as  to  the  appropriate  method  for  modeling  particular  aspects  of 
the  problem.  Nevertheless,  proper  statement  of  the  objective  must  be  approached 
as  a  critical  first  step  in  the  system  analysis/cost-effectiveness  process. 

After  obtaining  a  definitive  statement  of  the  system  objective  and  a  description 
of  the  system  boundaries,  the  specific  tasks  or  mission  profiles  are  generated. 
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To  properly  state  the  system's  objectives,  the  analyst  must  rely  heavily 
both  on  past  experience  gained  through  conducting  and  evaluating  similar 
studies  as  well  as  his  knowledge  of  the  military  environment  and  operation. 

There  are  three  pitfalls  encountered  in  specifying  system  objectives: 

(l)  the  analyst  may  state  the  objectives  too  broadly,  (2)  the  objectives 
may  be  limited  in  scope,  or  (3)  the  objectives  may  be  stated  in  such  a  man¬ 
ner  as  to  describe  a  particular  system  rather  than  a  functional  need.  If  the 
objectives  are  too  broad,  this  results  in  an  analysis  which  includes  an  im- 
practically  large  number  of  alternatives.  Additionally,  broad  objectives  often 
lead  to  a  large  number  of  assumptions  which  must  be  made  in  order  to  evaluate 
the  system(s ) . 

Generally,  as  the  number  of  assumptions  increase,  the  uncertainty  associa¬ 
ted  with  the  conculsions  reached  in  the  study  increases  proportionately.  If 
the  objectives  are  limited  in  scope,  good  alternative  systems  or  other  con¬ 
siderations  may  be  eliminated  from  the  analysis  which  could  result  in  a  less 
than  optimum  system  selection.  Sometimes  the  objectives  are  stated  in  terms 
of  a  specific  system  rather  than  a  functional  need.  This  undoubtedly  eliminates 
worthwhile  alternatives  from  consideration  and  biases  the  outcome  of  the  study. 

In  addition  to  establishing  reasonably  complete  boundaries  for  the  system, 
it  is  necessary  to  select  specific  factors  to  be  considered  in  the  analysis. 
Again,  this  task  cannot  be  defined  for  the  general  systems  analysis  or  cost- 
effectiveness  problem;  however,  general  assumptions  can  be  suggested,  and  these 
can  be  formalized  when  the  requirements  of  a  specific  analysis  are  given.  Typi¬ 
cal  assumptions  include  the  following: 

'  The  system  will  be  in  operation  for  T  years. 

■  All  external  factors  for  which  the  gross  cost  estimate  is  less  than 
of  the  estimated  cost  will  not  be  considered. 

'  Factors  with  relative-cost  estimates  between  and  x2$  will  be  included 
only  if  sample  data  and  knowledge  of  the  relationship  are  available  to 
yield  an  acceptable  degree  of  confidence  in  results. 

Factors  with  relative-cost  estimates  over  will  always  be  included, 
except  that  those  with  leverage  effects  will  be  excluded  if  their 
inclusion  would  entail  an  additional  level  of  analytical  complexity 
that  would  threaten  the  timeliness  of  the  analysis. 

All  assumptions  should  be  explicitly  stated  and  justified  by  factual  evi¬ 
dence.  If  none  exists,  the  reason  for  making  the  assumption  (e.g.,  mathematical 
convenience,  general  consensus)  should  be  stated  to  indicate  how  much  additional 
study  is  required  and  to  pinpoint  areas  where  errors  might  be  introduced. 
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The  constraints  placed  on  the  system  are  essentially  a  set  of  boundaries 
for  the  various  factors  within  which  the  solution  to  the  problem  must  be  found. 
They  may  include  a  fixed  budget,  a  period  of  time,  a  desired  effectiveness,  or 
a  method  of  operating  with  resources.  In  other  words,  the  constraints  asso¬ 
ciated  with  the  system  serve  to  designate  the  amount  of  freedom  allowed  in 
manipulating  system  variables  to  obtain  a  solution  to  the  problem. 

Constraints  can  often  be  used  as  the  sole  basis  for  distinguishing  between 
the  feasible  and  unfeasible  alternatives.  However,  a  constraint  can  be  so 
stringent  that  no  feasible  alternative  can  be  found,  and  thus  there  is  no 
solution  to  the  problem.  There  are  problems,  however,  for  which  a  solution 
must  be  obtained.  An  example  is  a  person  wiio  must  have  a  personal  automobile: 
His  constraints  are  that  the  cost  must  not  exceed  $2,000;  that  the  car  must  be 
new;  and  that  it  must  have  an  air  conditioner,  radio,  automatic  transmission, 
and  a  leather  interior.  There  is  no  solution  to  this  problem  within  these 
constraints.  If  the  person  must  have  a  car,  he  must  relax  one  or  more  of 
the  constraints  until  at  least  one  feasible  alternative  can  be  found. 

Thus  constraints  can  be  used  to  reduce  the  scope  of  the  problem,  but  they 
also  must  be  flexible  so  that  they  do  not  preclude  solution  of  the  problem. 

2.3  DEVELOP  MISSION  PROFILES 

After  the  requirements  and  objectives  have  been  formalized,  the  next  step 
is  for  the  analyst  to  develop  the  mission  description.  These  mission  descrip¬ 
tions  or  profiles  translate  the  requirements  and  objectives  of  the  system  into 
specific  statements  of  performance.  They  in  fact  describe  the  tasks  to  be 
performed  by  the  system  in  order  to  meet  the  stated  objectives.  Also  included 
in  the  mission  profile  are  such  considerations  as  threat,  environment,  and 
tactics . 

If  the  system  being  studied  were  a  transport  helicopter,  the  set  of  mission 
profiles  would  represent  all  of  the  tasks  that  the  transport  helicopter  was 
expected  to  accomplish.  Some  of  the  required  missions  might  be  of  greater 
importance  than  others  in  the  set  of  mission  profiles;  however,  this  possibil¬ 
ity  is  taken  into  account  in  the  presentation  of  results. 

The  generation  of  mission  profiles  can  be  characterized  by  the  flow  chart 
shown  in  Figure  2-2.  Initially,  the  operational  requirements  and  objectives 
are  determined  by  considering  such  factors  as  the  number  of  missions  required, 
the  functional  concepts  to  be  employed,  the  enemy  threat,  the  spectrum  of 
environments  that  may  be  encountered,  and  any  other  requirement  as  stated  in 
the  QMR.  These  requirements  and  objectives  are  then  translated  into  specific 
statements  of  performance  —  for  example,  the  number  and  type  of  communications 
equipments  required  for  a  mission,  the  information  rate  and  reliability  require¬ 
ments  of  the  communications  equipments,  and  the  range  and  weight  constraints. 
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FIGURE  2-2 

MISSION-PROFILE  DEVELOPMENT 


There  are  several  important  factors  that  must  be  considered  in  developing 
the  mission  profile: 

(1)  The  mission  parameters  (or  set  of  boundaries  of  the  mission)  should 
be  expressed  as  minimum  goals  and  maximum  values.  Care  must  be  exercised  to 
avoid  undue  rigidity  in  mission  parameters  and  each  mission  parameter  should 
be  assessed  as  to  its  criticality  in  terms  of  meeting  the  requirements  and 
objectives. 

(2)  Postulation  of  missions  is  not  solely  a  job  for  analysts,  rather  it 
should  be  dealt  with  on  a  team  approach  with  talent  drawn  from  both  the  tech¬ 
nical  and  military  communities. 

(3)  As  the  task  progresses  it  may  be  necessary  to  eliminate  certain  mis¬ 
sions  because  the  risks  associated  with  meeting  the  objectives  may  be  too 
high. 
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^4 )  The  missions  must  be  sufficiently  detailed  to  enable  the  analyst  to 
test  all  the  significant  system  parameters . 

(5)  The  missions  must  be  able  to  be  modeled  --  (war  gaming/scenarios ) . 

The  final  step  in  generating  the  mission  profiles  is  to  combine  all  the 
various  inputs  into  a  scenario  format  that  represents  the  combination  of  sys¬ 
tem  requirements,  threat,  environment,  and  tactics.  The  missions  thus  serve 
as  transfer  functions  which  relate  the  system  objective  to  the  performance  of 
the  alternative  systems. 

2.4  OBTAIN  CRITICAL  PERFORMANCE  PARAMETERS 

There  is  no  precise  procedure  for  defining  a  system  and  its  boundaries. 
Consequently,  all  possible  variables  within  the  system,  their  expected  inter¬ 
actions,  and  their  relevance  to'  the  problem  being  addressed  must  be  examined 
thoroughly. 

If  many  variables  are  considered  in  the  study,  a  valid  sensitivity  analysis 
can  be  made.  This  analysis  will  show  which  variables  are  critical  (and  may 
require  more  detailed  study)  and  which  variables  can  be  ignored. 

Not  all  performance  parameters  of  a  system  will  be  needed  in  the  evaluation 
of  a  system's  effectiveness,  because  some  performance  requirements  interface 
more  closely  with  the  system  objectives  than  do  others,  The  performance  re¬ 
quirements  of  the  AN/AFN-70A  (a  navigation  receiving  set)  are  a  good  example. 

The  general  description  of  the  AN/APN-70A  is  that  it  is  a  receiving  set  designed 
to  furnish  navigation  information  to  aircraft  up  to  distances  of  1500  nautical 
miles  from  special  ground  transmitters . 

Some  of  the  performance  requirements  listed  in  the  specification  for  the 
AN/APN-70A,  such  as  sensitivity,  selectivity,  accuracy,  and  stability,  are 
directly  associated  with  the  objective  of  the  receiver.  Others,  such  as  self¬ 
checking  function  and  oscillator  radiation,  may  not  be  sensitive  to  the  objec¬ 
tives  and  would,  therefore,  be  of  secondary  importance  in  the  effectiveness 
model.  However,  without  knowing  the  objectives  of  a  system,  it  would  be  diffi¬ 
cult  to  distinguish  precisely  which  performance  parameters  were  needed  to 
achieve  the  objective  and  which  were  not.  If  one  of  the  requirements  of  the 
APN-70  is  to  receive  signals  without  being  detected,  then  an  item  such  as 
oscillator  radiation  may  be  an  important  parameter.  Thus  it  would  appear  that 
that  to  talk  intelligently  about  the  importance  of  system-performance  parame¬ 
ters,  it  is  first  necessary  to  understand  which  of  the  many  parameters  of  a 
system  are  essential  for  the  objectives  of  the  system,  Examples  of  several 
performance  parameters  that  may  be  associated  with  different  types  of  equip¬ 
ment  are  as  follows : 


*  From  MIL-R-726B 
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Communications  ~  range,  noise  characteristics,  receiver  sensitivities, 
transmitter  power  outputs,  input  power  requirements,  jamming  vulnerability, 
and  number  of  channels. 

Computers  -  accuracy,  computational  speed,  input- output  formats,  memory' 
capacity,  retreval  speed,  language  capability,  and  analogue  to  digital  capa¬ 
bility. 

Antenna  -  gain,  coverage,  transmit  and  receiver  losses,  physical  limita¬ 
tions. 

Another  factor  that  must  be  considered  in  defining  the  critical  performance 
parameters  is  the  concept  of  controlled  and  uncontrolled  variables. 

Some  variables  can  be  controlled  early  in  the  system's  life.  For  example, 
the  time  needed  to  repair  a  failed  item  during  the  operational  phese  can  be 
controlled  to  some  extent  during  the  planning  and  design  phases.  Later,  during 
system  operation,  the  times  to  repair  are  distributed  according  to  some  observed 
function.  The  expense  of  changing  these  repair  times  is  generally  higher  in 
the  operational  phase  than  in  earlier  phases.  Other  variables  must  be  assumed 
to  be  distributed  according  to  an  assigned  or  predicted  function  during  the 
early  phases  of  life.  Later,  during  system  operation,  these  variables  can  be 
controlled.  Two  examples  of  variables  in  this  category  are  the  mission  times 
and  mission  frequencies  for  a  new  helicopter.  Still  other  variables  are  never 
subject  to  control  --  for  example,  the  weather;  and  there  are  some  variables 
that  can  always  be  controlled  --  for  example,  pay  scales. 

Some  variables  that  cannot  be  controlled  can  be  influenced  through  vari¬ 
ables  that  can  be  controlled.  A  common  example  is  the  set  of  variables  repre¬ 
senting  the  opposition's  reaction  to  changes  in  strategy  directed  at  them. 
Another  example  is  a  battlefield  war  game  in  which  player  A’s  strategy  includes 
influencing  player  B's  moves  through  the  control  A  has  over  his  own  forces. 

The  cost  parameters  will  depend  on  how  costs  have  been  defined  in  the  cost- 
effectiveness  problem.  They  may  be  time,  money,  lives,  distance,  or  area. 

The  criticality  of  cost  parameters  is  always  subject  to  change.  For  example, 
fuel  consumption  may  be  hardly  considered  until  a  certain  turn  of  events  limits 
the  quantity  available.  Hence,  a  military  mission  can  be  conceived  of  in  which 
the  cost  of  fuel  exceeds  the  cost  of  ammunition  even  though  the  dollar  costs 
of  the  two  are  the  same. 

2.5  SYNTHESIZE  ALTERNATIVE  SYSTEMS 

2.5.1  The  Total  System 

The  term  "System"  appears  to  defy  unique  definition;  i.e.,  one  man's  sys¬ 
tem  may  be  another  man's  component.  For  example,  a.  company  may  have  a  contract 
to  develop  a  radar  set.  Within  the  company  this  radar  may  be  thought  of  as  a 
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system  To  the  customer,  however,  this  same  radar  may  be  visualized  as  only 
one  element  of  a  larger  "system"  which  he  is  planning  for  the  control  of  air¬ 
craft  over  a  geographic  area.  This  latter  man's  system,  in  turn,  might  be  one 
part  of  a  much  larger  system  which  has  a  mission  of  managing  a  large  mass  air 
transportation  system.  There  is  really  nothing  wrong  with  any  one  of  these  per¬ 
sons  feeling  his  own  scope  of  design  is  a  system.  However,  a  definite  danger 
is  involved  in  not  recognizing  adequately  at  any  given  level  of  consideration 
the  relationship  of  the  given  system  to  all  potential  supersystems  and  the 
implied  interfaces. 

It  will  suffice  for  the  purpose  of  this  Guidebook  to  define  the  total  sys¬ 
tem  as  it  is  shown  in  Figure  2-3.  At  the  center  of  the  total  system  is  what 
is  termed  the  Object  System.  The  Object  System  may  be  defined  fragmentally  to 
coincide  with  the  scope  of  development  responsibility  which  a  prime  contractor 
would  normally  be  charged  with  Development  of  the  Object  System  must  be  accom¬ 
plished  in  consonance  with  the  requirements,  constraints,  and  interface  charac¬ 
teristics  of  the  world  in  which  it  will  ultimately  operate.  This  world  is 
categorized  into  four  blocks,  or  systems,  shown  in  the  figure. 


FIGURE  2-3 
THE  TOTAL  SYSTEM 


A  supporting  system  is  one  which  is  necessary  to  the  performance  of  the 
Object  System  but  not  in  the  direct  functional  line,  for  example,  a  maintenance 
system.  The  environment  which  would  tend  to  degrade  system  performance  (such 
as  the  physical  environment,  electromagnetic  environment,  and  others).  The 
demand  system  characterizes  the  need  for  which  the  Object  System  is  being 
developed.  In  a  military  situation,  the  demand  may  take  the  form  of  a  threat  - 
more  specifically,  it  might  be  the  approach  of  an  enemy  aircraft.  For  an 
Object  System  which  provides  service  (such  as  a  communications  system)  the 
demand  system  characterizes  the  potential  users  of  the  communications  service 
and  their  habits  and  technical  requirements.  Related  systems  are  those  with 
which  the  Object  System  must  cooperate  in  performing  its  intended  function. 
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Thus,  to  see  that  during  the  development  phase,  system  analysis  should  not 
be  confined  only  to  the  Object  System  but  should  be  used  to  characterize  the 
four  surrounding  system  categories. 

2.5-2  System  Configuration  Synthesis 

After  the  requirements  and  objectives  have  been  defined,  the  critical 
performance  parameters  obtained,  and  the  mission  profile  structured,  a  system 
or  set  of  alternative  systems  can  be  synthesized.  System  synthesis  requires 
several  ingredients,  among  which  the  more  important  are: 

'  An  adequate  understanding  of  the  mission  profile; 

'  An  appreciation  of  the  ultimate  user's  capabilities  and  limitations; 

'  Intuitive  judgment  concerning  the  effect  of  combinations  of  equipments 
operating  as  a  system; 

’  A  working  knowledge  of  the  state-of-the-art  capabilities  and  limitations 
in  technical  areas  from  which  the  potential  new  system  may  be  drawn;  and 

■  common  sense  to  seek  the  advice  of  experts,  in  questionable  or  critical 
technical  areas. 

2-5.2. 1  Basic  Functional  Subsystems 

The  first  step  past  a  mission  profile  definition  is  the  identification  of 
basic  functions  necessary  to  any  system  capable  of  accomplishing  the  mission. 
This  identification  should  be  carried  down  to  the  lowest  mission-oriented  level 
For  example,  the  functions  necessary  to  support  a  long-range  search  radar  are 
(l)power  supply,  (2)  antenna,  (3)  antenna  drive,  (4)  antenna  feeds  and  match¬ 
ing,  (5)  signal  formation  and  amplification,  (6)  single  reception  and  analysis 
elements,  (7)  transmit/receive  diflexing,  (8)  signal  processing,  and  (9)  sig¬ 
nal  display.  Each  of  the  above  functions  has  a  rather  direct  relationship  to 
the  mission  of  long-range  radar  search.  Figure  2-4  illustrates  a  similar  break 
down  for  a  ground-to-air  missile  system.  This  system  is  more  complex  than  the 
radar  example  and  in  turn  the  breakdown  is  not  carried  to  as  low  a  level,  of 
detail.  The  breakdown  in  Figure  2-4  is  shown  in  block  diagram  form  with  the 
connecting  lines  indicating  only  very  basic  relationships  between  blocks.  One 
should  be  careful  in  taking  this  first  step,  not  to  unintentionally  preclude 
any  particular  technical  approach  to  configuring  a  system. 

2. 5. 2. 2  Identify  Basic  Alternatives 

Once  the  basic  functional  elements  supporting  a  mission  have  been  defined, 
there  is  physical  realization  that  each  function  can  be  enumerated.  Table  2-1 
shows  such  an  enumeration  for  the  Zenith  missile  system.  The  left-hand  column 
of  the  Table  is  an  itemization  of  the  basic  functions  supporting  the  mission. 
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These  are  arranged  from  top  to  bottom  roughly  in  a  reversed  order  starting  from 
missile  detonation.  The  first  of  the  basic  categories  contains  the  configura¬ 


tion  options  which  could  conceivably  accomplish  the  function. 


Next,  i  s  a  column 


for  entering  design  factors  which  could  influence  the  selection  from  among  the 
options.  An  example  of  a  design  factor  is  warhead  weight.  To  accomplish  an 


equivalent  effect,  a  non-nuclear  warhead  would  involve  several  times  the  weight 
of  the  nuclear  warhead.  The  next  column  includes  factors  relating  to  the  use 
of  the  system  (such  things  as  side  effects  of  the  selected  options).  An 


example  of  a  related  mission  factor  can  again  be  given  regarding  the  choice 
of  warheads.  If  the  missile  is  to  be  detonated  over  or  near  friendly  forces. 


a  nuclear  warhead  has  certain  disadvantages  over  non-nuclear  warheads.  Some 
of  the  factors  entered  in  these  latter  two  columns  will  undoubtedly  be  the 


basis  for  eliminating  many  of  the  possible  configuration  options. 


Others  will 


simply  require  consideration  in  making  a  choice.  In  studying  Table  2-1,  certain 


FIGURE  2-4 


ZENITH  MISSILE  SYSTEM  -  FUNCTIONAL  DIAGRAM 
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TABLE  2-1 

BASIC  ALTERNATIVES  FOR  THE  ZENITH  SYSTEM 

Basic  Functions 

Options 

Design  Factors 

Related 

Mission  Factors 

Warhead 

Nuclear 

Miss  Distance  s  1000 
(la) 

Non-nuclear 

Miss  Distance  s  too 
(la) 

Use  in  all-out  or 
limited  Weir? 

Terminal  Guidance 
Sensor 

Optical 

Infrared 

RF 

Background  noise, 
sensor  response 
time 

Susceptible  to 
jamming 

Mid-Course 

Guidance  Sensor 

Radio  Link  to  Ground 
Radar 

Airborne  Radar 

Doppler 

Radar 

Inertial 

Position 

Missile  Control 
Technique 

Fins 

TVC  (Vanes) 

TVC  (liquid) 

Motor 

Solid 

Jet 

Throttleable 

Solid 

Liquid 

Missile  Computer 

Analog 

Digital 

Hybrid 

Missile  Fuzing 

Target  Doppler 

Intensity 

Missile  Arming 

Barometric  Pressure 

Radar  and  T/M  Link 

Guidance 

output 

Missile  Control 
Reference 


Inertial  Platform 


Strap-down 

Components 
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combinations  of  options  will  turn  out  to  be  impractical  from  the  interface 
standpoint.  For  example,  the  use  of  thrust  vector  control  alone  may  be  ruled 
out  if  the  terminal  flight  of  the  missile  is  to  be  unpowered. 

2. 5. 2. 3  Identify  Useful  System  Level  Performance  Parameters 

The  next  step  in  the  process  of  system  synthesis  is  to  define  a  hierarchy 
of  parameters  which  bridge  the  chasms  between  mission  profiles  and  functional 
subsystem  performance .  Figure  2-5  depicts  a  hierarchy  corresponding  to  the 
Zenith  system.  At  the  top  of  the  figure  are  two  basic  mission  performance 
parameters:  kill  probability  \(-%)  and  time  to  react  and  effect  a  kill.  The 

hierarchy  in  Figure  2-5  conceivably  could  be  continued  down  to  the  level  of 
module  and  even  piece-part  output  parameters.  It  is  not  necessary  to  do  so  at 
this  stage  of  development,  however,  since  the  lowest  levels  in  the  figure  repre¬ 
sent  parameters  which  easily  relate,  in  most  cases,  to  statements  about  state- 
of-the-art  and  gross  characteristics  of  design  options.  The  parameters  which 
are  underscored  with  heavy  lines  represent  what',  for  practical  purposes,  could 
be  called  system-level  performance  parameters.  Avoiding  the  question  of  what  is 
system  level  and  what  is  not,  suffice  it  to  say  that  the  underscored  parameters 
are  called  system  level  parameters  because  they  are  the  link  between  mission 
profile  statements  and  the  basic  functional  elements  as  defined  in  Table  2-1. 


mvn  TO  RBVT  AND  K11M  T  A  KjU 


FIGURE  2-5 

HIERARCHY  OF  PARAMETERS 
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2.5.3  State-of-the-Art  Analysis 

In  a  state-of-the-art  analysis,  each  critical  design  area,  or  component, 
should  be  viewed  from  three  general  points  of  view.  They  are  (l)the  technical 
suitability  of  a  proposed  technical  approach,  (2)  time  phase  consideration 
describing  all  of  the  particular  technical  approach  relating  to  the  mainstream 
development  time  frame,  and  (3)  the  economic  implications  of  incorporating  the 
technical  approach  into  the  system.  Table  2-2  is  illustrative  of  the  overall 
items  which  would  be  subject  of  scrutiny  in  determining  the  state-of-the-art 
feasibility  or  a  given  critical  design  area. 


TABLE  2-2 


CHARACTERISTICS  PERTINENT  TO  A  STATE-OF-THE-ART 
ANALYSIS  [ELECTRONIC) 


Dynamic  range 
Noise  threshold 
Saturation  limit 


Continuous 

Parameter 

Devices 


Linearity 
Sensitivity 
Frequency  response 
Gain 


Discrete 

Parameter 

Devices 


Isolation 

Stability 

Efficiency 

Hysteresis 

Pulse  rise  time  and  fall  time 

Overshoot 

Discriminat  ion 

Pulse  response  time 

Cycle  time 

Jitter 


Sensitivity  to  physical  environment 


I  Functional 
f  Characteristics 


Schock  and  vibration 
Temperature 

Humidity,  water,  ice,  snow,  rain,  fungus,  dust 


Radio  active  bombardment 

Sensitivity  to  electromagnetic  environment 

RFI 

Conducted  transients 
Inductive  coupling 
Capacitive  coupling 

Failure  modes 

Development  costs 
Evaluation  costs 
Production  costs 

Special  tools,  processes,  yield,  materials 

Installation  costs 
Maintenance  costs 
Support  costs 


Re  liability 
Characteristics 


L  Economic 
I  Characteristics 
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2.5.4  Conclusion 

The  major  competing  alternatives  must  be  identified  or  synthesized.  The 
variables  to  be  analyzed  for  quantification  of  these  alternatives  are  also 
identified.  Usually,  certain  alternatives  are  well  defined  at  the  beginning  of 
analysis.  One  of  the  major  benefits  of  an  analysis  is  that  it  can  generate  new 
alternatives  --  some  of  which  may  be  combinations  or  modifications  of  existing 
ones  --  as  the  analysis  progresses.  Initial  identification  and  selection  of 
variables  generally  require  screening,  e.g.,  through  preliminary  sensitivity 
analysis.  Decisions  on  alternative  and  variable  selection  should  be  continually 
re-examined  as  analysis  progresses. 

The  evolution  of  a  system  is  a  complex  process.  It  begins  with  an  idea  and 
proceeds  through  various  stages,  characterized  by  increases  in  knowledge  about 
the  system  until  it  reaches  a  stage  in  which  the  system  is  deployed.  With  each 
increase  in  knowledge  about  the  system,  some  decision  is  made  that  furthers  the 
system's  definition  and  reduces  the  degree  of  freedom  available  for  subsequent 
decisions.  At  each  stage  there  are  alternatives  that  can  have  a  significant 
effect  on  the  future  of  the  sys'tem.  Selection  of  one  of  these  alternatives 
results  in  a  refocusing  of  attention  to  a  new  set  of  alternatives. 

Different  alternatives  to  a  system  are  applicable  at  various  levels  of 
system  detail.  At  one  level,  when  action  is  necessary  in  response  to  a  threat, 
the  alternatives  may  include  the  procurement  of  a  new  missile,  artillery,  or 
aircraft.  These  are  evaluated  in  terms  of  defense  objectives  and  resource 
requirements.  Once  a  decision  is  made  on  one  of  these  alternatives,  say  the 
missile,  attention  is  focused  on  alternatives  within  the  missile  system.  These 
are  evaluated  in  terms  of  the  objectives  and  resources  applicable  to  the  missile. 
Selection  of  this  alternative  further  defines  the  system's  subsystems,  and 
attention  is  then  focused  on  alternatives  within  the  subsystem,  say  the  communi¬ 
cation  system.  This  process  continues  until  the  system  is  completely  defined. 

2.6  DEVELOP  HARDWARE  CHARACTERISTICS 

The  hardware  characteristics  for  each  alternative  system  are  determined  by 
the  necessity  for  meeting  specific  requirements  within  the  mission  profiles. 
These  hardwar?  characteristics  may  be  subject  to  intrs-system  trade-offs  to 
optimize  each  alternative.  (Trade-offs  will  be  discussed  in  section  2.12.)  por 
example,  the  size  of  a  helicopter  airframe  can  be  determined  by  the  requirement 
to  carry  a  given  number  of  troops.  The  horsepower  for  the  engines  can  now  be 
determined  by  considering  the  weight  of  the  airframe,  troops,  etc.,  and  the 
speed  and  hover  requirements  in  the  mission  profiles.  in  turn,  this  horsepower 
requirement  may  be  satisfied  with  a  single  engine  or  multiple  engines,  and 
thus  a  trade-off  to  determine  the  optimum  engine  configuration  is  desirable. 
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(Essentially,  a  single-engine  system  costs  less  in  maintenance  and  initial 
investment,  while  the  multiple-engine  is  safer  and  less  vunerable  to  enemy 
action. ) 

The  hardware  characteristics  for  a  communication  system  would  be  determined 
by  the  performance  requirements  (range,  channel  capacity,  system  reliability, 
etc.)  and  by  the  constraints  on  weight,  mobility,  etc. 

2.7  ESTABLISH  BASIS  FOR  EVALUATING  EFFECTIVENESS 

The  ultimate  output  of  any  system  is  the  performance  of  some  intended 
function.  This  function  is  frequently  called  the  mission  in  the  case  of  a 
weapons  system.  For  other  types  of  systems,  it  may  be  described  by  some  system- 
output  characteristic,  such  as  satisfactory  message  transmission  in  a  communica¬ 
tion  system  or  weather  identification  in  an  airborne  weather  radar.  The  term 
often  used  to  describe  the  overall  capability  of  a  system  to  accomplish  its 
mission  is  system  effectiveness.  A  more  precise  definition  of  this  expression 
will  be  given  later,  but  for  the  present  it  is  sufficient  to  observe  that  system 
effectiveness  is  related  to  that  property  of  system  output  which  was  the  basic 
reason  for  buying  the  system:  the  performance  of  some  intended  function.  If  the 
system  is  effective,  it  performs  this  function  well.  If  it  is  not  effective, 
attention  must  be  directed  to  the  system  attributes  that  are  deficient. 

Because  of  the  variety  of  systems  to  which  it  is  applied,  system  effective¬ 
ness  has  been  defined  in  a  number  of  ways.  The  WSEIAC*  effect  produced  the 
following  general  definition: 

System  Effectiveness  is  a  measure  of  the  extent  to  which  a  system  may 

be  expected  to  achieve  a  set  of  specific  mission  requirements.  It  is 

a  function  of  the  system's  availability,  dependability  and  capability. 

The  basic  approach  for  evaluating  the  effectiveness  of  a  system  can  be 
empirical-  or  analytic. 

The  empirical  approach  consists  Gf  collecting  data  and  evaluating  system 
effectiveness  by  observing  performance  characteristics  of  systems  in  the  field. 
However,  this  approach  is  applicable  only  for  systems  that  are  in  an  advanced 
phase  of  their  life  cycles. 

The  analytic  approach,  however,  does  not  require  that  the  system  be  in 
existence.  The  approach  is  based  on  the  construction  of  a  mathematical  model 
that  includes  predictions  of  system  characteristics  within  the  constraints 
imposed  by  the  analyst. 

The  Navy  System  Performance  Manual**  says  this  about  the  analytic  approach: 


*WSEIAC  --  Weapon  System  Effectiveness  Industry  Advisory  Committee. 

**NAVMAT  P3941. 
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"Purely  empirical  or  purely  analytic  methodologies  are,  of  course, 
not  very  useful.  The  former  yields  highly  authoritative  data  too 
late  to  be  useful,  while  the  latter  yields  answers  unsupported  by 
facts.  In  practice,  a  balance  is  sought.  This  balance  will  normally 
change  during  the  life  cycle  of  a  system.  As  data  about  the  behavior 
of  the  system  becomes  more  available,  the  analytic  model  gradually 
merges  into  an  empirical  model;  as  the  data  becomes  more  available 
and  as  confidence  in  their  value  increases,  statistical  sample  data 
supplant  the  assumptions.'1 

The  need  for  analytic  models  to  predict  system  effectiveness  is  based  on 
the  need  to  evaluate  the  effectiveness  of  a  system  before  it  has  been  in  the 
field  for  many  years.  Although  the  empirical  methods  are  required  to  provide 
inputs  to  the  systems  analysis/cost -effectiveness  process,  it  is  the  analytic 
approach  that  is  the  most  important  and  useful. 

One  analytic  effectiveness  model  that  has  generally  been  accepted  is  the 
WSE1AC  model.  The  WSE3AC  model  is  based  on  a  breakdown  of  the  effectiveness 
parameter  into  three  major  components  —  availability,  dependability,  and 
capability.  Availability  is  a  measure  of  the  system  condition  at  the  beginning 
of  a  mission;  dependability  is  a  measure  of  the  system  condition  during  the 
mission,  given  the  system  condition  at  the  start  of  the  mission;  and  capability 
is  the  ability  of  the  system  to  perform  its  mission,  given  the  system  condition 
during  the  mission. 

To  apply  the  model,  given  a  mission  definition  and  system  description,  it 
is  necessary  to  delineate  possible  mission  outcomes  and  significant  system 
states.  The  availability  and  dependability  measures  are  then  related  to  the 
possible  system  states,  and  the  capability  measure  relates  these  possible 
system  states  to  the  possible  mission  outcomes. 

For  the  very  simplest  of  cases,  in  which  a  system  must  be  in  either  a 
working  or  a  failed  state,  the  measures  of  availability,  dependability,  and 
capability  represent  the  following  fundamental  questions: 

*  Is  the  system  working  at  the  start  of  the  mission? 

■  If  it  is  working  at  the  start,  will  it  continue  working  throughout 
the  mission? 

’  If  the  system  works  throughout  the  mission,  will  it  actually  achieve 
mission  success? 

As  the  systems  considered  become  more  complex  —  e.g.,  when  there  are 
more  than  two  possible  systems  states,  and  such  factors  as  in-mission  repair, 
degraded  modes  of  operation,  multi-mission  requirements,  enemy  countermeasures, 
and  natural  environment  are  to  be  quantified  elements  in  the  model  —  these 
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questions  may  be  too  difficult  to  answer  with  simple  model  construction,  but 
they  still  represent  the  fundamental  WSEXAC  approach  towards  evaluating  effec¬ 
tiveness  on  a  mission-oriented  basis. 

The  general  framework  for  the  analysis  of  system  effectiveness  is  given 
in  Figure  2-6.  The  elements  of  system  effectiveness  are  discussed  briefly  in 
Sections  2.7.1  through  2.7.3,  and  are  outlined  in  Figure  2-6a. 
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ELEMENTS  OF  EFFECTIVENESS 
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The  concept  of  availability  concerns  the  system's  condition  at  the  start 
of  a  mission.  The  WSEIAC  definition  is  as  follows: 

Availability  is  a  measure  of  the  system  condition  at  the  start 
of  a  mission.  It  is  a  function  of’  the  relationships  among 
hardware,  personnel,  and  procedures. 

Availability  has  also  been  expressed  as  the  probability  that  the  system 
is  operating  satisfactorily  at  any  time,  when  used  under  stated  conditions. 

For  this  guide  book,  the  more  general  WSEIAC  definition  is  used. 

Application  of  the  availability  concept  requires  clear  definition  of  what 
is  included  in  the  system  and  of  the  system's  mission.  Availability  is  a  specific 
measure;  it  is  therefore  usually  necessary  to  define  more  than  one  system  and 
its  associated  mission  and  to  define  availability  for  each  case. 

In  estimating  a  system's  availability,  care  must  be  taken  to  consider  how 
the  "system"  has  been  defined  and  bounded.  For  example,  with  regard  to  a  group 
of  2f  tanks,  an  analyst  may  be  equally  interested  in  the  availability  of  a 
single  tank,  several  tanks,  and  the  group  of  tanks.  For  the  25  tanks,  then, 
several  systems  and  several  corresponding  missions  can  be  defined,  and  the 
measure  of  availability  will  be  different  in  each  case. 

Assume  that  the  system  is  defined  as  a  group  of  four  tanks  (from  the  25) 
and  that  the  mission  is  to  attack  as  a  group.  If  at  the  start  of  a  mission  one 
tank  is  found  to  be  inoperative,  the  system  will  be  counted  as  being  in  an  un¬ 
available  state.  However,  the  group  mission  is  not  necessarily  aborted,  since 
it  might  be  possible  to  draw  a  replacement  from  another  group,  in  this  case, 
then,  availability  is  considered  zero,  and  yet  it  is  not  necessary  to  abort  the 
mission. 

On  the  other  hand,  if  the  system  consists  of  all  2C  tanks  and  the  system's 
mission  is  to  provide  tanks  for  specific  missions  as  required,  availability 
might  be  a  measure  of  the  number  of  tanks  available  for  assignment  at  any  time. 

It  is  not  a  measure  of  the  number  of  operational  tanks,  since  availability  mat' 
be  zero  —  for  example,  when  80  percent  of  the  tanks  are  performing  specific 
missions  and  20  percent  are  inoperative.  In  this  case,  it  is  also  possible  to 
conceive  of  an  availability  of  zero  when  all  tanks  in  the  group  are  operational 
and  performing  their  assigned  missions. 

In  another  situation  the  measure  of  interest  may  be  the  number  of  tanks 
that  are  operational  at  any  time.  The  system  is  the  entire  group  of  tanks,  and 
the  system  mission  is  to  maintain  all  tanks  in  operational  status.  \n  this  case. 
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it  is  possible  that  availability  will  be  100  percent  and  yet  a  specific  mission 
will  be  aborted  because  all  tanks  are  already  performing  specific  missions. 

2.7.2  Dependability 

The  second  major  element  in  the  system-effectiveness  framework  is  depend¬ 
ability.  The  WSEIAC  definition  is  as  follows: 

Dependability  is  a  measure  of  the  system  condition  at  one  or 
more  points  during  the  mission,  given  the  system  condition  at 
the  start  of  the  mission. 

The  nature  of  this  concept  is  similar  to  what  is  commonly  referred  t  0  as 
reliability,  except  that  reliability  is  usually  defined  as  the  probability 
that  a  system  will  perform  satisfactorily  for  at  least  a  given  period  of  time 
when  used  under  stated  conditions.  The  more  general  WSEIAC  definition  is  used 
herein  since  the  system' s  maintainability  characteristics  also  influence  its 
dependability. 

As  with  the  concept  of  availability,  application  of  the  dependability 
concept  requires  an  exact  statement  of  the  system's  composition  and  mission. 

A  single  estimate  of  dependability  for  the  system  in  question  may  not 
convey  all  that  should  be  known  about  it  and  related  systems.  For  instance, 
when  it  is  said  that  the  probability  of  hitting  a  target  with  a  weapon  from  an 
attack  helicopter  is  0.90,  it  is  implied  that  a  favorable  result  will  be  obtain¬ 
ed,  with  probability  0.90.  This  statement  also  indicates  that  there  is  a  0.10 
probability  that  the  target  will  not  be  hit,  but  does  not  give  any  information 
about  the  causes  or  consequences  of  such  failure.  Did  the  helicopter  crash 
before  reaching  the  target?  Did  the  weapon  stray  into  a  friendly  area?  Did 
the  weapon  fail  t  o  release  from  the  helicopter?  Was  the  helicopter  grounded 
because  of  bad  weather?  To  evaluate  the  system  properly,  it  is  necessary  to 
consider  the  circumstances  surrounding  a  failure. 

When  the  probabilistic  definition  of  reliability  first  came  into  general 
use  several  years  ago,  one  of  the  most  important  prerequisites  to  applying  it 
was  that  "failure"  be  explicitly  defined.  This  'is  implicit  in  the  definition, 
since  to  say  what  is  satisfactory  performance  under  given  conditions,  it  is 
necessary  to  state  carefully  what  is  not  satisfactory  performance  (i.e.,  failure) 
under  those  same  conditions. 

The  problems  become  more  complex  in  the  case  of  multiple  systems  and 
missions.  Interest  may  be  centered  on  the  dependability  of  a  single  commu¬ 
nications  system,  an  equipment  in  the  system,  or  a  group  of  systems.  Each 
system  may  be  capable  of  a  number  of  different  missions,  some  more  critical 
than  others.  Combining  the  dependability  figures  for  all  systems  and  missions 
tends  to  obscure  the  favorable  or  unfavorable  characteristics  of  an  individual 
system  or  mission.  On  the  other  hand,  if  attention  is  focused  on  a  single 
system  or  mission,  the  tendency  is  to  ignore  the  favorable  or  unfavorable 
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characteristics  of  the  related  systems  and  mission.  Measurement  of  depend¬ 
ability  requires  identification  not  only  of  each  critical  system  and  mission 
but  also  of  the  implications  of  success  or  failure  in  each  instance.  Then 
the  several  estimates  of  dependability  are  displayed  in  the  form  of  a  vector. 
Although  this  type  of  display  does  not  facilitate  the  decision-making  process 
(if  anything,  it  makes  the  process  more  difficult)  ,  it  does  lead  to  more  ac¬ 
curate  decision-making. 

2.7.3  Capability 

The  third  major  element  in  the  system-effectiveness  concept,  capability , 
is  defined  in  the  WSEIAC  report  as  follows: 

Capability  is  a  measure  of  the  system's  ability  to  achieve 
the  mission  objectives,  given  the  system  condition  during  the 
mission.  Capability  specifically  accounts  for  the  performance 
spectrum  of  the  system. 

A  similar  concept  that  expresses  this  characteristic  of  a  system  in 
probabilistic  terms  is  Design  Adequacy,  which  has  been  defined  as  the  probability 
that  a  system  will  successfully  accomplish  its  mission,  given  that  the  system 
is  operating  within  design  specifications.  In  this  guide  book,  the  more  general 
WSEIAC  definition  is  used,  but  in  specific  instances  system  capability  may  be 
expressed  as  a  probability.  It  should  be  noted  that  capability  or  design  adequacy 
is  not  solely  an  inherent  characteristic  of  system  hardware.  Capability  depends 
to  a  significant  degree  on  the  mission  assigned  to  the  system.  A  system  that 
was  designed  to  accomplish  a  specific  task  may  very  well  have  a  high  capability 
for  that  task.  (In  probabilistic  terms,  perhaps  the  Design  Adequacy  is  unity.) 
However,  if  it  is  used  for  a  more  difficult  or  complex  mission,  its  capability 
for  this  new  mission  may  be  low. 

The  measurement  and  prediction  of  system  capability  is  a  rather  complex 
problem  in  itself.  The  difficulties  introduced  by  multi-modal  systems  and 
multiple  missions  in  the  application  of  availability  and  dependability  concepts 
are  also  present  in  the  use  of  the  capability  concept.  Further,  the  capability 
concept  has  not  yet  been  developed  to  the  extent  that  it  can  be  quantified  by 
standard  techniques. 

The  system  analyst  must  be  particularly  careful  to  distinguish  clearly 
between  capability  and  dependability.  It  was  mentioned  earlier  that  failure 
and  the  circumstances  surrounding  failure  must  be  explicitly  defined  before 
the  dependability  concept  can  be  applied.  The  same  is  true  of  the  capability 
concept.  Consider,  for  example,  a  l/^-ton-truck  tire  blow-out  that  occurs  at 
60  mph,  on  a  hot  day  (110°F),  upon  impact  of  the  tire  with  a  jagged  hole  in 
the  pavement.  Is  the  failure  attributed  to  a  lack  of  dependability  or  cap¬ 
ability?  If  the  tire  (system)v/as  designed  for  high-speed  and  high-temperature 
environments,  and  to  withstand  high  impact  loads,  then  the  blowout  (failure)  is 
attributed  to  lack  of  dependability  (reliability)  since  the  conditions  of 
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satisfactory  operation  were  defined  to  include  these  severe  environments.  Qi 
the  other  hand,  if  the  blowout  occurs  on  a  tire  designed  to  operate  in  much  less 
severe  environments  (perhaps  40  mph  at  80°P),  then  the  failure  is  attributed 
to  lack  of  capability.*  In  the  first  case  the  tire  (system)  had  adequate  cap¬ 
ability,  but  its  dependability  was  low.  In  the  second  case  the  system's 
dependability  may  have  been  high,  but  its  capability  (for  the  particular  mis¬ 
sion)  was  less  than  adequate.  In  either  case,  it  is  important  to  note  that  the 
system  effectiveness  of  the  tire  is  below  the  acceptable  level. 

2.8  DEFINE  MEASURES  OF  EFFECTIVENESS 

The  measures  of  effectiveness**  for  some  systems  are  easy  to  obtain  and 
commonly  accepted  --  such  as  ton-miles/day  for  a  transportation  system,  and 
single-shot  kill  probability  for  an  anti-tank  missile.  However,  for  other 
systems,  including  many  electronics  systems,  ^no  overall  measures  of  effective¬ 
ness  have  been  developed.  For  example,  the  effectiveness  of  a  communication 
system  may  be  measured  in  terms  of  information  rate,  information  reliability, 
system  reliability,  and  system  availability;  however,  it  is  not  possible  to 
combine  these  four  factors  into  one  overall  measure  of  effectiveness.  This  is 
not  necessarily  a  disadvantage,  because  the  decision-maker  can  still  make  a 
choice  even  if  the  effectiveness  is  presented  as  four  separate  numbers;  of 
course,  the  choice  may  not  be  as  simple  or  clear-cut. 

The  measures  of  effectiveness  are  subject  to  change  with  time  in  a  battle¬ 
field  situation.  At  one  time,  effectiveness  may  be  measured  in  terms  of  the 
damage  inflicted  on  enemy  supply  routes.  At  another  time,  it  may  be  measured 
in  terms  of  how  long  it  takes  to  intercept  an  intruding  tank  column. 

If  the  mission  profiles  have  been  specifically  defined,  the  problem  of 
defining  measures  of  effectiveness  is  greatly  reduced  in  that  the  effectiveness 
can  now  be  expressed  as  a  probability  of  accomplishing  all  or  a  given  part  of 
the  system  mission.  In  other  cases,  where  a  set  of  specific  mission  profiles 
cannot  be  obtained,  the  effectiveness  must  be  related  to  the  physical  character¬ 
istics  of  the  system  --  for  example,  range,  channel  capacity,  speed,  etc. 

in  the  operational  situation,  attributing  unsatisfactory  performance  to  a 
lack  of  either  capability  or  dependability  is  often  based  on  the  frequency 
of  unsatisfactory  performance.  Frequent  failure  in  normal  operation  makes 
capability  suspect,  while  infrequent  failure  makes  dependability  suspect. 

Also  called  criteria  for  effectiveness. 

t  Much  work  is  currently  under  way  in  this  area  to  define  measures  of  effec¬ 
tiveness. 
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2.9  FORMULATE  MODELS 

2,9*1  General 

A  system  model  is  essentially  a  mathematical,  logical,  or  physical  repre¬ 
sentation  of  the  interdependencies  between  the  objectives  and  the  resources 
associated  with  the  system  and  its  use.  For  dealing  with  the  effectiveness  of 
complex  systems,  the  model  is  usually  in  the  form  of  mathematical  equations 
(mathematical  model)  or  computer  programs  for  simulating  system  operation 
(simulation  model) ,  or  both. 

On  the  assumption  that  a  set  of  system  objectives  has  been  translated  into 
an  optimization  criterion,  the  model  builder  is  required,  minimally,  to  con¬ 
struct  a  model  that  will  enable  quantification  of  the  critical  effectiveness 
and  cost  parameters  as  a  function  of  the  resource  variables. 

The  overall  cost-effectiveness  model  is  usually  one  that  consists  of  several 
sub-models,  each  of  which  may  be  based  on  models  at  still  lower  levels.  Figure 
2-7  indicates  one  means  for  sub-model  classification.  It  should  be  noted  that 
there  are  many  other  schemes  for  classifying  models. 

There  is,  naturally,  a  great  deal  of  interaction  among  the  sub-models,  and 
model  integration  is  required  in  the  same  sense  that  system  integration  is 
required. 

Constructing  sub-models  (and  integrating  them  into  an  overall  model)  is, 
for  most  "real  world"  situations,  still  more  of  an  art  than  a  science,  largely 
because  the  validity  of  the  model  cannot  be  tested  through  controlled  experi¬ 
mentation;  thus  the  collaboration  of  people  with  wide  experience  in  the  areas 
of  concern  is  an  important  requirement. 


FIGURE  2-7 

A  SUBMODEL  CLASSIFICATION  SCHEME 


2-25 


AMCP  706-191 

2.9.2  Assumptions 


All  assumptions  required  for  the  model  should  be  explicitly  stated  and.  if 
possible,  supported  by  factual  evidence.  If  no  such  evidence  exists,  it  is 
advisable  to  state  the  reason  for  the  assumption,  e.g.,  mathematical  simplicity 
or  consensus  of  opinion,  in  order  to  indicate  the  degree  to  which  the  assump¬ 
tions  will  require  further  justification  and  to  pinpoint  the  areas  in  which 
errors  might  be  introduced. 

2. 9-3  Adequacy 

A  model  must  be  adequate  in  the  sense  that  all  major  variables  to  which 
the  solution  is  sensitive  are  quantitatively  considered  where  possible.  Many 
of  these  variables  will  have  been  preselected.  Through  manipulation  of  the 
model,  some  of  the  variables  may  be  excluded  or  restricted,  and  others  may  be 
introduced.  Non- quantifiable  variables  must  be  accounted  for  by  modification 
of  the  solution  rather  than  by  direct  incorporation  into  the  model.  In  this 
sense  they  are  quantifiable. 

2.9.^  Representativeness 

Although  no  model  can  completely  duplicate  the  "real  world",  it  is  required 
that  the  model  reasonably  represent  the  true  situation.  For  complex  problems, 
this  may  be  possible  only  for  sub-parts  of  the  problem,  which  must  be  pieced 
together  through  appropriate  modeling  techniques.  As  an  example,  analytic 
representation  may  be  possible  for  various  phases  of  a  complex  maintenance 
activity.  The  outputs  from  these  analyses  may  then  be  used  as  inputs  to  a 
simulation  procedure  for  modeling  the  complete  maintenance  process. 

2.9.5  Uncertainty 

The  various  types  of  uncertainties  involved  in  the  problem  cannot  be  ig¬ 
nored,  nor  can  they  be  "assumed"  out;  they  must  be  faced  squarely.  There  may 
be  technological  uncertainties  involved  with  some  of  the  system  alternatives, 
operational  uncertainties  involved  with  planning  and  carrying  out  the  mission, 
uncertainties  about  enemy  strategy  and  action,  and  statistical  uncertainties 
governed  by  the  laws  of  chance.  The  simplest  approach  is  to  make  "best  guesses", 
but  this  may  lead  to  disastrous  results,  since  the  probability  of  guessing 
correctly  for  every  uncertainty  is  quite  small.  For  cases  involving  statistical 
uncertainty,  functions-of-random-variables  theory  or  such  procedures  as  Monte 
Carlo  techniques  may  be  used.  For  the  other  types  of  uncertainties,  the  general 
approach  is  to  examine  all  major  contingencies  and  compute  resultant  cost- 
effectiveness  parameters.  The  optimization  criterion,  then,  must  be  adaptable 
for  use  in  the  evaluation  of  the  set  of  cost  effectiveness  results.  The  devel¬ 
opments  of  decision  theory  and  game  theory  become  most  applicable  in  the  selec¬ 
tion  of  a  decision  model  in  these  cases,  since  different  alternatives  may  be 
best  for  different  contingencies. 
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2.9.6  Data 

The  availability  of  relevant  data  plays  an  important  role  in  the  develop¬ 
ment  and  application  of  a  model.  Data  are  required  to  support  assumptions, 
select  alternatives,  and  define  constraints,  as  well  as  to  define  the  cost  and 
effectiveness  constants  in  the  proposed  model.  Since  missing  data  may  prevent 
valid  model  application,  the  model  builder  should  investigate  this  possibility 
early  in  the  model  development  stages  and  plan  to  obtain  missing  data  or  ad¬ 
just  the  model  accordingly.  If  a  great  expenditure  of  time  and  money  is  re¬ 
quired  to  obtain  the  necessary  data,  the  analyst  may  be  forced  to  weigh  the 
risks  of  using  what  is  available  (and  making  necessary  assumptions)  against  the 
value  received  in  return  for  the  costs  of  the  data- collection  and  analysis 
effort . 


2.9.7  V  aliditv 


The  final  test  of  the  model  is  whether  or  not  it  yields  the  best  system. 
Unfortunately,  this  determination  can  be  made  only  after  systems  are  developed 
and  in  use,  if  it  can  be  made  at  all.  However,,  certain  questions  will  disclose 
weaknesses  that  can  be  corrected: 

(1)  Consistency  -  are  results  consistent  when  major  parameters  are 

varied? 


(2)  Sensitivity 

(3)  Plausibility 

(4)  Criticality 

(5)  Workability 

(6)  Suitability 


do  input-variable  changes  result  in  output  changes 
that  arc  consistent  with  expectations? 

arc  results  plausible  for  special  cases  .where  prior 
information  exists? 

do  minor  changes  in  assumptions  result  in  major 
changes  in  the  results? 

does  the  model  require  inputs  or  computational 
capabilities  that  are  not  available  within  the 
research  bounds?. 

is  the  model  consistent  with  the  objectives;  i.e., 
will  it  answer  the  right  questions? 


2.9.8  Types  of  Models 


There  are  several  types  of  models,  including  mathematical  models,  gaming 
models,  simulation  models,  and  operational  exercises.  These  models  are 
described  below. 


2.9.8. l  Mathematical  Models 

Mathematical  models  are  characterized  by  the  exclusive  use  of  equations 
to  represent  the  characteristics  of  the  system  and  thus  are  the  most  abstract 
of  the  four  categories.  The  basis  for  such  equations  can  range  from  pure 
hypothesis  to  the  analysis  of  available  and  relevant  data.  Mathematical  models 
generally  provide  a  great  deal  of  flexibility,  but  often  at  the  expense  of 
simplifying  the  real-world  situation  to  develop  a  usable  model  and  thereby 
decreasing  the  representativeness  of  the  model. 
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2.9. 8.2  Gaming  Models 

In  the  gaming-model  technique,  operations  and  resource  usage  are  simulated 
through  scale  models,  computer  programs,  or  physical  analogs.  Personnel  in 
operational  decision-making  capacities  are  participants  as  an  integral  part  of 
the  model.  Examples  of  gaming  models  are  military  war  games  and  the  use  of 
aircraft  simulators. 

2. 9. 8. 3  Simulation  Models 

In  the  simulation-model  technique,  all  aspects  of  the  system,  its  resource 
usage,  and  its  operations  are  simulated  in  an  abstract  form,  usually  through 
computer  programs.  The  basic  operational  flow  is  structured  and  probabilistic 
paths  are  determined  through  appropriate  random-variable  generating  procedures. 
Such  computer  models  are  popularly  called  Monte  Carlo  procedures.  ^  example 
is  the  simulation  of  component  failure  and  repair  times  to  provide  estimates 
of  system  availability  and  maintenance  and  logistic  requirements. 

2. 9. 8. 4  Operational  Exercise 

In  the  operational  exercise,  actual  system  resources  are  used,  generally 
in  a  simulated  operating  environment.  Examples  are  a  controlled  experiment  of 
weapon  firings  involving  military  personnel  and  resources,  and  a  military  field 
maneuver  between  red  and  blue  forces.  The  costs  of  such  exercises  are  generally 
high,  and  thus  the  number  and  extent  of  the  trials  must  necessarily  be  limited. 
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2.10  GENERATE  EFFECTIVENESS  EQUATIONS 

The  basic  WSEIAC  system-effectiveness  equation  is  the  product  of  an 
availability  vector,  a  dependability  matrix,  and  a  capability  vector,  which 
are  defined  as  follows: 


A  =  j^a^,  a2'  t*le  availability  vector 

a.  =  probability  system  is  in  state  i  at  the  beginning 
of  the  mission 


the  dependability  matrix 


d^  ,  =  probability  of  a  system-state  transition  from  state  i  to 
J  stage  j  over  a  fixed  period  of  time 


C 


the  capability  vector 


=  the  capability  of  the  system  for  performing  the  mission, 
cj  given  the  system  is  in  state  J 

A  typical  term  of  the  product  is  A  DC 

ai  dij  °,1 

where 

is  the  probability  that  the  system  is  in  state  i  at  the  beginning  of 
the  mission 

d^j  is  the  probability  that  the  system  will  make  the  transition  from  state 
i  to  state  j  over  a  fixed  time  period 

Cj  is  the  probability  (or  expected  value  associated  with  mission 
accomplishment)  that  the  system  can  perform  its  mission,  given  state  j 

It  is  emphasized  that  the  WSEIAC  Model  is  not  a  self-contained,  directly 
applicable  mathematical  equation  for  effectiveness.  As  stated  may  times  in 
several  ways  in  the  WSEIAC  Task  Group  II  report,  the  "model"  is  actually  a 
framework  for  effectiveness  quantification  --  a  basic  routine  for  constructing 
an  appropriate  model.  Although  the  model  framework,  represented  by  the 
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product  A  D  C,  can  in  some  instances  be  used  directly,  this  simple  product 
will  not  work  for  a  particular  system-mission  combination.  It  was  not  intended 
that  this  product  be  always  directly  applicable  —  only  that  the  elements  of 
availability,  dependability,  and  capability  be  incorporated  in  such  a  manner 
that  the  model  framework  could  be  applied. 


As  a  simple  example,  the  product  .A  D  (3  in  actuality  is  based  on  the 
assumption  that  mission  performance  is  evaluated  at  a  single  point  in  time  — 
the  end  of  the  mission.  For  many  cases,  this  is  not  reasonable,  if  D  (t  ,  t  ) 
is  defined  to  be  the  dependability  matrix  over  the  time  interval  ( t  ,  ts),  and 
if  the  Markov  assumption  holds ,  i,e,, 


£  (V  V 


£  (V  V 


for  all  such  that  tr<t^<t  ,  then  the  effectiveness  0f  the 
is  represented  by 


system  at  time  t. 


E(tk)  =  A 


D  (0, 


If  the  mission  is  one  in  which  continuous  performance  is  required  over  the 
mission  length  t  ,  the  effectiveness  of  the  system,  assuming  well-behaved  func¬ 
tions,  may  possibly  be  quantified  as  the  time  average  of  E(tk)  —  that  is, 

.  tm 


-i  / 

t  Jo 


E(t )  d  t 


Note  that  if  at  each  performance  time  the  capability  co-efficient  c,  equals 
one,  and  if  state  J  belongs  to  the  set  of  satisfactory  states  and  is  zero  other¬ 
wise,  the  above  equation  for  E  reduces  to  the  expected  fraction  of  the  mission 
performance  time  that  the  system  is  in  a  satisfactory  state. 

An  extension  to  the  WSEIAC  methodology  is  necessary  if  the  Markov  assumption 
does  not  hold.  In  this  case,  the  capability  matrix  must  be  written  as  an  NxN 
Matrix  (N  =  number  of  system  states),  with  an  entry  for  each  state  transition. 

Exhibit  1  presents  a  system-effectiveness  problem,  with ’its  solution,  that 
illustrates  the  application  of  this  technique. 
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EXHIBIT  1 

EXAMPLE  OF  SYSTEM-EFFECTIVENESS  PROBLEM 


Two  communications  systems,  A  and  B.  are  used  simultane¬ 
ously  to  transmit  information.  Should  either  of  the  systems 
fail,  the  remaining  one  is  capable  of  transmitting  alone  (A 
and  B  are  statistically  independent).  Failures  in  either  (or 
both)  systems  are  not  repaired  during  a  transmission  period, 
but  are  repaired  during  a  period  when  the  equipments  are 
normally  shut  down. 

A  transmission  will  be  started  whenever  at  least  one  of 
the  systems  is  available  (in  other  words,  it  is  not  necessary 
that  both  A  and  B  be  in  operable  condition  in  order  to  start 
a  transmission). 


The  respective  mean  failure  times,  mean  repair  times,  and 
bit  rates  for  A  and  B  are  given  below: 


System 

Mean  Failure  Time,  f 

Mean 
Repair 
Time,  H 

Transmission  Rate,  r 

A 

12  hours  (exponential) 

S  hours 

120,000  bits/hour 

B 

24  hours  (exponential) 

6  hours 

100,000  bits/hour 

A  normal  transmission  period  consists  of  3  uninterrupted 
hours. 
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EXHIBIT  1 


QUESTIONS 


QUESTION  1 

What  is  the  effectiveness  of  A  and  B  combined,  if 
effectiveness  is  defined  as  the  probability  of  transmitting 
at  least  300,000  bits  during  a  normal  transmission  period? 

QUESTION  2 

What  is  the  effectiveness  of  A  and  B  combined,  if 
effectiveness  is  defined  as  the  expected  (average)  number  of 
bits  transmitted  during  a  normal  transmission  period? 

QUESTION  3 

What  is  the  answer  to  Question  1  if  both  values  of  T 
are  increased  50#? 

QUESTION  4 

What  is  the  answer  to  Question  2  if  both  values  of  T?  are  in¬ 
creased  50#? 

QUESTION  5 

What  is  the  answer  to  Question  1  if,  instead  of  changing 
the  values  of  T,  the  values  of  R  are  both  decreased  50#? 

QUESTION  6 

What  is  the  answer  to  Question  2  if,  instead  of  changing  the 
values  of  T,  the  values  of  R  are  both  decreased  50#? 


For  all  auestlons.  the  svstem  state 
designation  will  be: 


Cxi  nCigwroU  ion 

A  •  B 
A  •  B 
A  •  B 
A  •  S 


State  Number 

1 

2 

3 

4 
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EXHIBIT  1 

DISCUSSIONS  AND  ANSWERS 


NOTE:  All  calculations  rounded  off 
to  2  places  as  they  occur. 


QUESTION  1 

Availability  Calculations:  The  availability  (A)  of  a  system  is  the  probability 
that  a  system  is  operating  at  any  point  in  time  and  is  given  by  the  equation: 


T  +  R 


In  particular,  the  availability  of  subsystems  AA  and  Ag  are  as  follows: 


A  =  12 

A  12  +  8 


o.6o 


ab  ~  °‘80 

Definition:  a.±  =  p( state  i  exists  at  start  of  transmission  --  a  function 

of  the  availabilities  of  subsystems  A  and  B): 

a1  =  (Aa)  (Ab)*  =  (0.60)  (0.80)=  0.48 

a2  =  (Aa)  (1-Ab)  =  (0.60)  (0.20)=  0.12 

a3  =  (1-Aa)  (Ab)  =  (0.40)  (0.80)=  0.32 

a4  =  (1-Aa)  (1-Ag)  =  (0.40)  (0.20)=  0.08 

where  AA  and  Ag  are  the  availabilities  of  subsystems  A  and  B,  respectively. 

A  Matrix:  A=  [  0.48  0.12  0.32  0.08] 

Dependability  Calculations:  The  measure  of  dependability  that  will  be  used  in 
this  example  is  the  reliability  measure  associated  with  the  operation  of  subsystems 
A  and  B.  Reliability,  then,  is  defined  as  the  probability  that  a  system  will 
satisfactorily  perform  its  functions  for  a  given  period  of  time.  Because  elec¬ 
tronic  systems  are  being  considered  in  this  example,  the  reliability  function  is 

_  t 

assumed  to  be  exponential  and  is  given  by  the  equation  R(t)  =  e  ^  where  t  is 
the  mission  time.  Thus  the  reliabilities  for  Subsystems  A  and  B  are  as  follows: 

-  -  -3- 

ReliabilityA  (3  hours)  =  e-  ^A  =  e  12  =  e-  °-25  _  0.78 

_  t 

Reliabilityg  (3  hours)  =  e  _  £-  _  £-  0.125  =  q  88 

Reference  Multiplication  Law.  p.  4-5  of  this  guidebook. 
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EXHIBIT  l 


Elemental 

Transition  Probabilities. 

p(A 

a)3 

=  0.78 

p(B 

-b)3 

=  0.88 

p(A 

=  0.22 

p(B 

-S)3 

=  0.12 

p(X 

-a)3 

=  O 

P(B 

-b)3 

=  0 

p(T 

=  1 

P(S 

-B), 

=  l 

Definition:  =  p(going  from  state  i  to  state  j),  hours 


State  Transition  Probabilities: 


dn  =  (0.78)  (0.88)  =  0.69  d31  =  (0)  (0.88)  =  0 

d12  =  (0.78)  (0.12)  =  0.09  d32  =  (0)  (0.12)  =  0 

d13  =  (0.22)  (0.88)  =  0.19  d33  =  (1)(0.88)  =  0.88 

dl4  =  (0.22)  (0.12)  =  0.03  dj4  =  (1)(0.12)  =  0.12 


d21 

(0.78) 

(0) 

=  0 

d4l 

-  (0) 

(0) 

=  0 

dgg  = 

(0.78) 

(1) 

=  O.78 

CVJ 

=  (0) 

(1) 

=  0 

^3  = 

(0.22) 

(O) 

=  0 

d43 

=  (1) 

(0) 

c?  0 

^4  = 

(0.22) 

(1) 

=  0.22 

d44 

=  (1) 

(1) 

=  1 

0.69 

0.09 

0.19 

0.03 

0 

0.78 

0 

0.22 

D  Matrix: 

°  =  0 

0 

0.88 

0.12 

0 

0 

0 

1 

Capability  Calculations 


Definition:  ^  =  p (transmitting  s  300,000  bits  under 

the  state  transition:  (i  —  J)3J. 


11 

1 

C31 

=  0 

12  = 

1 

OJ 

CO 

0 

=  0 

13  = 

1 

C33 

=  1 

T4  = 

(see  below) 

c34 

=  0 

21  = 

0 

c4l 

=  0 

22  = 

1 

c42 

=  0 

;23  = 

0 

c43 

=  0 

24  = 

(see  below) 

c44 

=  0 
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EXHIBIT  1 

For  and  c consider  notational  solution: 

.  Transmission  rates:  rA,  (constant)  •  Mission  time:  T 
.  Failure  rates:  XA,  (exponential)  •  Conditions:  rAT  >  P;  rgT  =  p 
.  Bits  to  be  transmitted:  p 


Then 


dtA  = 


1 


l-e 


^P 

rA  -V 

e  -e  R 


(1) 


=  0.15  (for  the  numbers  involved) 


r  T 


- IXIT- 


dt 


l-e 


"B 


'B 


p-rAfcA 

rB 


(l-e  -Xat)  (l-e-V) 


=  0.55  (for  the  numbers  involved) 


(NOTE:  Equation  2  is  very  sensitive  to  round-off 
errors:  7  place  entries  are  needed  to 
ensure  2  place  accuracy  of  result) 


C±J  Matrix:  cij 


1  1  1  0.55 

Oio  0.15 
0  0  1  0 

0  0  0  0 
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EXHIBIT  1 


Effectiveness  Calculations 

The  effectiveness  equation  is  given  by: 


E  = 


4  4 

Z  Xai 


i=i  j=i 


which  may  be  written: 


4 


J-l 


E  -  A.  x 


Z  d2J 

j=l 

4 

Z  d3J  °3J 
1=1 


(3) 


4 

Z  d*J  °4J 

=  1 


4 

Z  dlj  clj  “  (°*69^  (1)  +  (°-°9)  (1)  +  (0.19)  (1)  +  (0.03)  (0.55)  =  0.99 
J=l 
4 

Z  *2 J  °2 j  =  (°)  (0)  +  (°-78)  0-)  +  (°)  (°)  +(0.22)  (0.15)  =  0.81 
J-l 

4 

Z  d3J  °3J  =  +  <0)  +  (°'88)  (1)  +  (°'12)  (0)  =  °"88 

j-l 


V  d4j  c4j  =  (0)  (0)  +  (0)  (0)  +  (0)  (0)  +  (1)  (0)  =  0 

J-l 


and 


E  =  [  0.48  0.12  0.32  0.08  ]  x 

=  0.48  +  0.10  +  0.28  +  0  =  0.86  =  Answer 


§:§? 

0.88 

0 
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EXHIBIT  1 


QUESTION  2 

Commenl 

The  only  change  from  Question  1  occurs  in  the  methodology,  definitions,  and 
calculations  related  to  the  C^j  Matrix. 

Definition:  T|  <  T  is  the  mean  failure  time  of  a  system 
subject  to  the  condition  that  is  known  to 
have  failed  by  time  T. 


For  an  exponential  system  with  failure  rate  M 


*1 


w 


Thus,  the  conditional  mean  failure  times  for  A  and  B  are: 

Ta|  <  T  =  1.4  hours 

(for  the  numbers  involved) 

*B|  <  T  =  1.5  hours 


Definition:  Fj  <  T  =  the  expected  number  of  bits 

transmitted  by  a  system  which  is  known 
to  have  failed  by  time  T. 


It  follows  directly  that 

F|  <  T  =  (T|  <  T)  r  (5) 

Thus,  for  systems  A  and  B, 

PA|  <  3  =  (120,000)  (1,4)  =  168,000  bits 

PB|  <  3  =  (100,000)  (1.5)  =  150,000  bits 


Definition:  P|  2  T  =  expected  number  of  bite 

transmitted  by  a  system  which  has  not 
failed  by  time  T 
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It  follows  directly  that 


so  that 


0j  §  T  =  rT 

*aI  *  3  =  (120,000)  (3)  =  360,000  bits 
PB|  s  3  =  (100,000)  (3)  =  300,000  bits 


EXHIBIT  1 

(6) 


Definition: 


=  expected  number  of  bits 
transmitted  under  the  transition: 


(1  -  J)3 


C11  = 

360,000 

+ 

300,000 

= 

660,000 

C3l 

= 

0 

+ 

0 

= 

0 

C12  = 

360,000 

+ 

150,000 

= 

510,000 

C32 

= 

0 

+ 

0 

= 

0 

C13  = 

168,000 

+ 

300,000 

= 

468,000 

C33 

= 

0 

+ 

300,000 

= 

300,000 

Cl4  = 

168,000 

+ 

150,000 

= 

318,000 

C34 

= 

0 

+ 

1 50,000 

= 

150,000 

C21  - 

0 

+ 

0 

= 

0 

4 

= 

0 

+ 

0 

= 

0 

II 

CVJ 

CVJ 

0 

360,000 

+ 

0 

= 

360,000 

°42 

= 

0 

+ 

0 

= 

0 

c23  = 

0 

+ 

0 

= 

0 

C43 

= 

0 

+ 

0 

= 

0 

II 

CVJ 

0 

168,000 

+ 

0 

s 

168,000 

c44 

= 

0 

+ 

0 

= 

0 

C.  .  Matrix:  c, 
—J.  -J-J 


660,000  510,000  468,000  318,0001 

0  360,000  0  168,000 

0  0  300,000  150,000 

0 


Effectiveness  Calculations 

Equation  3  still  holds,  and  the  D  Matrix  is  unchanged: 

4 


dlJ  clj  =  (0*69H66°,000)+(0.09)(5l0,000)+(0.19)(468,000)+(0.03)(3l8,000)=599,760 


4 

^  d2J  C2J  =  (°)  (°)  +  (0.78)  (360,000)  +  (0)  (0)  +  (0.22)  (168,000)  =  317,760 
.1=1 
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EXHIBIT  1 


4 

X  d3J  °33  =  (0)  (0>  +  (0)  +  (°-88)  (300,000)  +  (0.12)  (150,000)  =  282,000 

j=l 


4 

X  d4j  c4j  =  (0)  (0)  +  (0)  (0)  +  (0)  (0)  +  (1)  (0)  =  ° 
J=1 


and  E  =  [  0.48  0.12  0.32'  0.08  ]  x 


599,760 

282,(760 

282,000 

0 


=  287,885  +  38,131  +  90,240  +  0  =  416.256  bits  =  Answer 


QUESTION  3 

Comments 

.  50^  increase  in  T  values  results  in:  TA  =  18;  TB  =  36 
•  The  methodology  is  identical  to  that  of  Question  1. 

Availability  Calculations 

AvailabilityA  =  =  0.69 

Availabilityg  =  =  0.86 

A  Matrix:  A  =  [  0.59  O.IO  0.27  0.04  ] 

Dependability  Calculations  o 

ReliabllityA  (3  hours)  =  e  1  '  =  e  °*187  _  o.84 

3 

Reliability^  (3  hours)  =  e  ^  =  e  ^=0.92 


0.07 

0.15 

0.01 

O.W 

0.84 

0 

0. 

D  Matrix:  D 

0 

0 

0.92 

0.08 

0 

0 

0 

1 

0 
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EXHIBIT  1 


Capability  Calculations 
Comment : 


All  c^j  values,  except  for  and  are  identical  to  those  in 
Question  1. 

From  Equation  1,  Cg^  =  0.19 
From  Equation  2.  =  0.56 


Matrix:  5l i 


Effectiveness  Calculations 
From  Equation  3, 


E  =  [  0.59  0.10  0.27  0.04  ] 


0  0 


1  0.56 

0  0.19 

1  0 
0 


0  0 


-  rounded  off 

from  0.996 

1 

0.87 


0.92 

L  °  J 

=  0.59  +  0.09  +  O.25  +  0  =  0. 93  =  Answer 


QUESTION  4 

Comments 

.  50^  increase  in  T  values  results  in:  T.  =  18;  w  =  36 
.  A  and  D  Matrices  are  those  of  Question  3 
•  Cjj  Matrix  methodology  is  identical  to  that  of  Question  2 

Capability  Calculations 
From  Equation  5, 

P A |  <  3  =  (120,000)  (1.5)  =  180,000 
0B|  <  3  =  (100,000)  (1.5)  =  150,000 


fii  Matrix: 


660,000 

510,000 

480.000 

330,000  1 

0 

360,000 

0 

180,000  1 

0 

300,000 

150,000 

L  0 

0 

0 

O 
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Effecth  eness  Calculations 


From  Equation  3, 


E  =  [  0.59  0.10  0.27  0.04  ]  x 


619,200 

331,200 

288,000 

0 


=  365,328  +  33,120  +  77,760  =  476.208  bits  =  Answer 


QUESTION  5 

Comments  ; 

.  50$  decrease  in  R  values  results  in:  TfA  =  4j  Eg  =  3 

.  The  only  change  from  Question  1  occurs  in  the  calculation  of  the  A  Matrix 
•  The  methodology  is  identical  to  that  of  Question  1 


Availability  Calculations 


AvailabilityA  =  =  0.75 

oh 

Availability  g  =  57  =  0.89 

A  Matrix:  A  =  (  0.67  0.08  0.22  0.03  ) 

Effectiveness  Calculations 

E  =  (  0.67  0.08  0.22  0.03  )  x 


0.99 

0.8l 

0.88 

0 


=  0.92  =  Answer 


QUESTION  6 


Comments 


.  50^ 
.  The 

•  The 

•  The 


decrease  in 
only  change 
methodology 
A  Matrix  is 


R  values  resu Its  In:  ft  =  4;  =  3 

from  Question  2  occurs  in  the  evaluation  of  the  A  Matrix, 
is  identical  to  that  of  Question  2 
identical  to  that  calculated  in  Question  5 
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EXHIBIT  1 


Effectiveness  Calculations 


E  =  C  0.67  0.08  0.22  0.03  ]  x 


599.760 

317.760 
317,760 

t-  vt.  f  WV 
0 


401,839  +  25,421  +  62,040  =  489,300  bits  -  Answer 


SUMMARY 


T 

A 

T 

iB 

*A 

E1 

p(  3  300,000) 

Average  Bits 

12 

24 

8 

6 

0.86 

416,256 

18 

36 

8 

6 

0.93 

476,208 

12 

24 

4 

3 

0.92 

489,300 
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The  method  followed  in  any  cost  prediction  is  straightforward  enough  but  is 
apt  to  be  quite  laborious.  Furthermore,  the  data  on  which  any  prediction  must  be 
based  are  difficult  to  collect,  and  the  gross  estimates  that  it  is  necessary  to 
employ  must  be  treated  with  a  good  deal  of  reserve.  To  make  any  cost  prediction 
at  all,  it  is  necessary  (l)to  break  the  expenditures  down  into  rather  small 
categories,  (2)  to  collect  as  much  past  experience  on  expenditures  in  each  category 
as  possible,  and  (3)  to  predict  from  this  information  how  much  is  likely  to  be 
spent  in  each  category  for  the  project  being  costed.  Thereafter,  all  the  cate¬ 
gories  must  again  be  combined  to  obtain  the  system  cost  as  a  function  of  time. 

2.11.2  The  RAND  Method 

Many  agencies,  both  in  the  DoD  and  industry,  are  performing  military  cost 
analysis  and  developing  costing  methodology.  The  RAND  Corporation  has  been  one 
of  the  leaders  in  the  costing  field.  The  major  costing  concepts  proposed  by  the 
rand  Corporation  are  as  follows: 

•  Categorization  of  costs  into  research  and  development,  initial  investment, 
and  operating  costs 

.  Use  of  individual-system  costing  and  total- force- structure  costing 

.  Use  of  incremental  costing 

.  Concentration  on  most  important  cost  factors 

These  concepts  are  described  briefly  below,  and  several  of  them  arc  discussed 
In  more  detail  in  the  following  sections. 

The  categorization  of  costs  into  research  and  development,  initial  investment, 
and  operating  costs  is  consistent  with  the  DoD  programming  system.  Some  advantages 
of  this  categorization  are  that  the  time  phasing  of  the  costs  are  readily  apparent, 
the  total  lifetime  cost  for  alternative  system  lifetimes  is  easily  obtained,  and 
the  impact  that  changes  in  the  research,  development,  and  initial  investment  costs 
have  on  operating  costs  can  be  observed  and  traded  off. 

Total-force- structure  costing  is  much  more  involved  than  individual-system 
costing.  Individual-system  costing  does  not  examine  the  interactions  between 
itself  and  other  systems  in  the  total  force.  This  makes  the  cost  analyst's  task 
simpler,  and  is  particularly  useful  in  costing  future  systems  (where  interactions 
with  other  systems  are  not  well  defined  anyway).  Total-force-structure  costing 
examines  the  cost  of  a  system  in  the  framework  of  the  total  force.  This  requires 
Information  on  interactions  among  the  systems  in  the  total  force,  and  also  cost 
data  for  the  total  force. 
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Incremental  costing  is  an  approach  that  determines  the  change  in  cost 
associated  with  achieving  some  change  in  effectiveness.  If  a  decision  is  made 
today  to  develop  a  new  system,  incremental  costing  is  used  to  determine  the  cost 
to  develop  that  system  starting  from  today.  Costs  that  have  been  incurred 
previously  are  not  counted,  and  the  costs  for  existing  equipment  and  facilities 
that  can  be  utilized  in  the  new  system  are  not  counted.  In  most  cases,  incre¬ 
mental  costing  is  the  type  of  costing  that  decision-makers  are  asking  for  when 
they  say,  "What  will  I  be  getting  for  uy  money?" 

A  sensitivity  analysis  consists  of  changing  each  of  the  variables  in  the 
study  in  turn,  while  ths  remaining  variables  are  held  constant,  to  determine  how 
small  changes  in  each  variable  can  affect  the  study  result. 

A  cost- sensitivity  analysis  is  normally  used  to  determine  which  parameters 
have  the  greatest  impact  on  the  total  cost.  The  cost  analyst  can  then  concentrate 
his  efforts  on  the  most  important  cost  factors. 

2.11.2.1  Cost  Categorization 

Systsn  costs  have  been  categorized  in  a  number  of  ways,  depending  to  a  great 
extent  on  the  type  and  applicability  of  available  data.  The  objective  in  any  of 
these  categorizations  is  to  focus  attention  on  the  major  resources  that  will  be 
consumed  during  the  life  of  the  system.  Information  on  resources  is  produced 
that  can  be  compared  with  information  on  available  resources;  alternative  courses 
of  action  can  be  evaluated  according  to  the  amount  of  resource  consumption  they 
involve. 

The  military  grouping  of  costs  corresponds  to  the  program  phases  in  which 
the  costs  are  incurred: 

.  Research  and  Development  Costs.  All  the  costs  necessary  to  bring  a  system 
into  readiness  for  introduction  into  active  inventory. 

.  Initial  Investment  Costs.  All  costs  incurred  in  phasing  a  system  into  the 
operational  force.  They  include  the  costs  of  procurement  of  prime  and 
special  equipment,  facility  construction,  personnel  training,  and  procure¬ 
ment  of  initial  spares. 

•  Operating  Costs,  All  costs  necessary  to  the  operation  of  the  system  once 
it  has  been  phased  into  the  operational  inventory.  While  both  R&D  and 
investment  costs  are  incurred  just  once,  the  operating  costs  continue 
throughout  the  life  of  the  system. 

The  curves  of  Figure  2-8  show  typical  distributions  of  these  costs  over  the 
life  cycle  of  a  system.  Further  subdivisions  of  these  costs  are  shown  in  the 
following  paragraphs. 
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FIGURE  2-8 

SYSTEM  COST  CATEGORIES 


Examples  of  the  types  of  costs  in  each  major  category  are  as  follows: 
Research  and  Development  Costs 
.  Design  and  development 

Preliminary  research  and  design  studies 

Development  engineering  and  hardware  fabrication 

Development  instrumentation 

Captive  test  operations 

Fuels,  propellants,  and  gases 

Industrial  facilities 

.  System  test 

Test-vehicle  fabrication 

Vehicle  spares 

Test  operations 

Test  ground  support  equipment 

Test  facilities 

Test  instrumentation 

Fuels,  propellants,  and  gases 

Data  reduction  and  analysis 

Maintenance,  supply,  miscellaneous 

•  System  management  and  technical  direction 

Initial  Investment  Costs 
.  Installations 

Construction  of  new  building,  airfields,  etc. 
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•  Equipment 

P  r  i  m  a  r  y  -  m  i  s  s  i  o  n  equipment 
Specialized  equipment 
Other  equipment 

■  Stocks 

Initial  allowances 
Maintenance  float 
Equipment  spares  and  spare  parts 
Combat  consumption  stocks 
Ammunition 

.  Initial  training 

.  Miscellaneous  investment 

Initial  transportation  of  equipment  and  spares 
Initial  travel 

Init i al  propellants,  oils,  and  lubricants 
Operating  Costs 

•  Equipment  and  installations  replacement 

Primary -miss ion  equipment 
Specialized  equipment 
Other  equipment 
Installations 

•  Training 

•  Pay  and  allowances 

.  Propellants,  oils,  and  lubricants 

Primary  mission  equipment 

Other  propellants,  oils,  and  lubricants 

.  Services  and  miscellaneous 

Transportation 

Travel 

Other  services  and  miscellaneous 

•  Nondirect  administrative  and  support  costs 

2.11.2.2  Costing  Individual  Systems  and  Total  Force  Structure 

A  military  system  can  normally  be  defined  by  describing  three  key  elements 
(l)the  mission,  with  threat  and  environment;  (2)  the  method  of  operation;  and 
(3)  a  description  of  the  physical  makeup  of  the  system  and  its  support  system. 
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If  these  three  elements  are  defined,  and  there  are  no  interrelationships 
with  other  systems  that  must  be  considered,  then  the  type  of  cost  analysis  that 
is  normally  used  is  the  individual  system-cost  analysis. 

Individual  system-cost  analysis  is  less  involved  and  requires  fewer  data 
than  would  be  necessary  if  the  system  were  not  isolated  from  the  total  force. 

The  difference  between  individual-system  and  total- force- structure  cost 
analyses  is  the  level  at  which  the  analysis  is  carried  out.  The  total-force- 
structure  cost  analysis  is  a  higher-level  analysis  and  includes  many  individual 
systems  with  the  associated  interactions  among  the  individual  systems. 

There  are  cases  in  which  an  individual  system  cannot  be  costed  realistically 
without  the  use  of  a  total- force- structure  cost  analysis.  For  example,  the 
development  of  a  new  anti-aircraft  weapon  would  require  a  total-force- structure 
cost  analysis.  The  entire  air-defense  capability  could  be  costed  (l)with,  and 
(2)  without  the  new  individual  system.  The  difference  in  cost  between  (l)and 
(2)  would  be  the  cost  of  the  new  anti-aircraft  weapon. 

2.11.2.3  Incremental  Costing 

Incremental  costing  accounts  for  additional  costs  associated  with  the 
additional  effectiveness  of  a  new  system.  The  major  factors  included  in  the 
concept  of  incremental  costing  are  inherited  assets,  sunk  costs,  and  salvage 
value. 

Inherited  assets  arc  those  existing  equipments,  existing  facilities,  and 
trained  personnel  that  are  available  for  the  new  system.  Inherited  assets  arc 
not  included  in  the  cost  comparison  for  alternative  systems.  For  example,  if  a 
new  radar  system  can  utilize  existing  repair  installations,  the  initial  cost  of 
these  installations  is  not  included  in  the  cost  analysis;  i.e.,  the  installations 
are  free. 

Sunk  costs  are  those  costs  that  have  been  expended  prior  to  a  given  decision 
point,  in  time,  and  these  costs  are  not  included  in  a  cost  comparison.  For 
example,  suppose  two  alternative  communications  systems  are  being  considered  for 
development,  and  one  system  lias  already  incurred  $2  million  of  R&D  funds,  while 
for  the  other  alternative  system  (possibly  an  improved  version  of  an  existing 
system)  no  R&D  fund  has  been  expended.  The  decision  to  select  either  alternative 
may  be  based  naturally  on  cost,  effectiveness,  technical  freasibility,  time 
scheduling,  etc.  However,  in  costing  each  alternative,  the  $2  million  that  has 
already  been  spent  on  the  first  alternative  is  a  sunk  cost,  and  therefore  is  not 
included  in  the  cost  comparison  --  i.e.,  no  matter  which  alternative  is  selected, 
an  additional  Slmillion  of  R&D  funds  will  be  expended. 
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Another  factor  included  in  the  incremental  costing  concept  is  salvage  value. 
Salvage  value  takes  into  account  the  cost  saving  that  may  be  realized  from  selling 
or  transferring  a  system  to  a  future  organization  when  the  system  is  phased  out, 
or  the  cost  saving  from  selling  the  system  as  scrap. 

2.11.3  Cost-Estimating;  Relationships 

The  most  important  tool  available  to  the  cost  analyst  is  the  cost-estimating 
relationship  (CER).  These  provide  a  method  of  predicting  the  cost  of  a  new 
system.  These  relationships  are  developed  by  collecting  accumulated  cost  data 
on  similar  systems  and  correlating  such  costs  to  appropriate  characteristics  of 
the  new  system  (weight,  size,  number  of  parts,  etc.).  For  some  types  of  systems 
that  have  been  in  existence  for  many  years,  such  as  airframes,  enough  data  have 
been  generated  to  permit  development  of  CERs  for  use  on  new  airframes;  however, 
for  more  advanced  systems  that  incorporate  a  substantial  number  of  state-of-the- 
art  improvements,  currently  available  cost- prediction  techniques  do  not  provide 
the  needed  accuracy. 

The  standard  method  for  developing  cost-estimating  relationships  is  through 
the  use  of  multiple-regression  analysis.  To  use  the  multiple-regression  approach, 
a  general  assumption  is  made  that  the  dependent  variable  —  in  this  case,  a  cost 
category  --  is  related  to  the  predictor  variables  by  a  linear  equation  of  the 
following  form: 

fo(C>  =  bo  +  blfl(xll5  x12’  •  ■  Xr1l^  +  b2f2(x123  x123  •  •  •■»  xrg2^ 

+  .  .  .  +  bnfn(xln,  x2n,  .  .  *rnn) 


where 

fQ(C)  is  a  function  of  the  cost 

is  the  jth  prediction  parameter  in  the  Cth  subset 
fj,  f2,  .  .  f  are  functions  of  the  x's 
bo’  bl3  '  ‘  ■’  bn  are  comPuted  regression  coefficients 

Although  the  general  function  is  linear  with  respect  to  the  regression 
coefficients,  it  is  not  necessarily  linear  with  respect  to  C  or  the  x's.  Thus, 
for  example,  an  equation  of  the  form 


transforms  to  the  required  linear  form 

log  C  =  log  a  +  log  +  b2  X2 


Although  nonlinear  equations  can  be  used  through  various  mathematical  curve¬ 
fitting  procedures,  the  advantages  of  using  a  least- squares  analysis  based  on 
linear  equations  in  the  coefficients  and  the  flexibility  of  such  linear  forms 
strongly  favors  the  standard  regression  approach. 
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Normally,  separate  CERs  are  developed  for  each  of  the  major  cost  categories 
(Research  and  Development,  Initial  Investment,  and  Operating  Costs).  Factors  that 
influence  one  cost  category  may  have  little  effect  on  another  category  or  may  even 
have  an  opposite  effect.  For  example,  an  equipment  ultra- high-reliability  program, 
such  as  that  currently  being  sponsored  by  DoD  through  the  three  services,  will 
increase  development  costs  but,  it  is  hoped,  greatly  reduce  operational  costs. 
Therefore,  parameters  relating  to  reliability  will  have  a  positive  development- 
cost  relationship  but  a  negative  operational-cost  relationship.  It  would  be 
unwise  to  combine  the  two  cost  categories  into  one  CER  since  accuracy  will  probably 
be  diminished;  equally  important,  trade-off  aspects  will  be  lost. 

It  is  desirable  to  limit  the  final  number  of  parameters  in  the  CERs  to 
minimize  use  difficulty  as  well  as  to  maximize  the  degrees  of  freedom  in  the 
analysis . 

2.11.4  Problems  in  Cost  Analysis 

2.11.4.1  Cost  Canmensurablllty 

There  are  two  main  techniques  for  making  costs  commensurable:  amortization 
and  discounting.  Amortization  is  the  spreading  of  the  system  research  and  develop¬ 
ment  and  initial  investment  costs  over  the  lifetime  of  the  system.  Discounting 
is  used  to  reflect  the  greater  value  of  present  money  over  future  money,  because 
of  the  possibility  of  investing  present  money  for  a  gain. 

Amortization  and  discounting  are  not  normally  used  for  military  costing, 
for  the  following  reasons: 

•  Goverment  directives  call  for  yearly  estimates  of  actual  expenditures; 
this  lnforma,tlon  aids  in  the  preparation  of  the  annual  budget. 

.  The  discounting  rate  is  difficult  to  determine;  it  lias  been  estimated  to 
be  from  1  percent  to  20  percent. 

.  These  factors  are  normally  insensitive  as  compared  with  other  costing 
considerations. 

At  present,  the  discounting  rate  normally  used  in  Army  studies  is  15  percent. 

2.11.4.2  Cost  Unccrtaint\ 

Every  cost  estimate  is  uncertain,  fran  the  initial  component-cost  estimate 
up  through  the  aggregation  of  system  costs.  Cost  analysts  have  been  careful  to 
differentiate  between  this  type  of  uncertainty  in  the  cost  estimates  (cost¬ 
estimating  uncertainty)  and  uncertainties  in  what  exactly  is  to  be  costed  (require¬ 
ments  uncertainty).  In  the  latter  case,  a  detailed  description  of  the  inter¬ 
relationships  associated  with  the  system  may  consume  as  much  as  75  percent  of  the 
total  project  time.  The  most  serious  errors  in  the  cost  analysis  can  usually  be 
traced  to  the  assumptions  and  interrelationships  on  which  the  cost  estimates  are 
based. 
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Cost-estimating  uncertainty  Is  the  statistical  uncertainty  caused  by  errors 
in  the  cost  data,  inaccurate  cost-estimating  relationships,  and  differences  in 
the  cost- analysis  approach. 

The  analyst  should  develop  not  only  the  cost  estimates,  but  also  an  indication 
of  the  confidence  level  or  possible  ranges  of  the  costs.  Use  of  the  cost-sensi¬ 
tivity  analysis  is  also  desirable  to  show  the  Impact  that  uncertainty  can  have  on 
the  final  cost  estimates. 

2.11.4.3  Data  Collection 

One  of  the  greatest  problems  in  cost  analysis  is  obtaining  sufficient  data 
and  accurate  data.  The  basic  data  compiled  to  support  the  requirements  of  the 
cost  analysis  should  meet  the  following  requirements: 

.  They  should  be  collected  in  sufficient  quantity  to  provide  significant 
sample  sizes  of  the  various  system  characteristics  and  cost  parameters 
being  studied.  The  confidence  in  results  increases  with  the  quantity  of 
observations.  Accordingly,  every  effort  must  be  made  to  acquire  sufficient 
data  from  actual  surveillance  of  systems  in  an  operational  environment. 

If,  however,  adequate  data  of  this  type  are  not  available,  it  may  be 
necessary  to  resort  to  estimating  techniques.  Several  proven  techniques 
are  available  for  various  equipments. 

.  They  should  reflect  current  system  conditions.  Timely  collection  of  input 
data  is  required  if  the  cost  analysis  is  to  depict  current  conditions  in 
the  system.  Many  diverse  factors  affect  the  cost  of  developing,  purchasing, 
operating,  and  maintaining  a  modern  system.  Unfortunately,  at  least  from 
a  cost-development  standpoint,  almost  all  of  these  factors  are  dynamic. 

.  They  should  be  accurate.  The  Importance  of  using  the  most  accurate  data 
available  cannot  be  overemphasized.  The  stringent  requirement  for  accurate 
data  is  related  to  the  intrinsic  nature  of  the  mathematical  approach.  Many 
compound  summing  operations  (or  multiplications)  will  be  accomplished 
during  the  cost  analysis;  thus  any  inaccuracies  in  the  data  will  also  be 
compounded  during  these  mathematical  manipulations.. 

.  They  should  be  representative  of  the  operational  situation  of  interest. 

The  system-cost  characteristics  are  known  to  be  affected  by  the  operational 
and  maintenance  environment.  Until  such  time  as  the  direction  and  extent 
of  the  various  Influencing  factors  upon  the  system  are  more  explicitly 
defined,  it  will  be  desirable  to  collect  data  from  the  specific  operational 
situation  in  the  cost  analysis. 
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2.11.5  Cost-Analvsis  Application 

2.11.5.1  Problem  Formulation 

The  specific  form  that  any  cost  analysis  will  take  depends  on  the  particular 
system  being  studied.  However,  in  general,  there  are  three  factors  that  must  be 
considered  in  every  costing  problem:  (l)the  costing  methods  that  will  be 
employed,  (2)  the  type  of  data  that  will  be  required,  and  (3)  the  sources  of  the 
data.  These  factors  are  discussed  below. 

The  costing  method  employed  is  normally  one  of  three  general  types  --  a 
catalogue  price,  a  cost-estimating  relationship,  or  an  estimate  based  on  a  similar 
system.  The  catalogue  price  is  used  where  the  component  is  an  off-the-shelf  item. 
The  cost- estimating  relationship  can  be,  developed  on  the  basis  of  design  and  per¬ 
formance  characteristics,  or  previous  cost.  The  estimate  of  cost  based  on  a 
similar  system  is  used  when  the  system  being  anaflyzed  is  sufficiently  similar  to 
an  existing  system  that  a  valid  cost  analogy  can  be  made. 

The  type  of  data  that  is  required  for  a  cost  analysis  may  be  categorized  into 
the  assumptions  and  constraints,  the  description  of  the  system,  and  the  cost  in¬ 
formation.  As  an  example  of  the  types  of  data  required  for  a  cost  analysis,  assume 
that  the  avionics  on  a  group  of  helicopters  is  maintained  by  a  dual  maintenance 
organization,  consisting  of  organizational  maintenance  and  direct  support  mainte¬ 
nance. 

If  a  complaint  against  an  avionic  system  is  received,  organizational  personnel 
try  to  verify  the  complaint;  if  they  verify  it,  they  (l)perform  some  maintenance 
at  the  helicopter  (this  may  consist  of  changing  a  black  box)  and  (2)  in  a  certain 
percentage  of  the  cases,  generate  some  direct  support  maintenance. 

If  they  do  not  verify  the  complaint,  they  have,  of  course,  spent  some  time 
in  the  investigation;  however,  that  complaint  is  disposed  of. 

The  following  data  are  required  in  the  cost  analysis  of  this  system: 

•  Number  of  complaints  per  month 

•  Fraction  verified 

.  Manpower  needed  to  verify 

.  Fraction  disposed  of  at  the  helicopter 

•  Manpower  needed  to  dispose  of  the  complaint  at  the  helicopter 

.  Cost  of  the  bits  and  pieces  needed  at  the  helicopter 

.  Manpower  needed  to  provide  the  bits  and  pieces  at  the  helicopter 

•  Manpower  needed  to  replace  the  black  box 

.  Cost  of  the  replacement  black  box 

From  "Reliability  Engineering"  ,  copvrighted  15)64  by  ARINC  Research  Corporation. 
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•  Manpower  needed  to  obtain  the  black  box 

.  Time  delay  before  the  faulty  box  is  repaired  and  ready  for  use 
.  Time  needed  to  repair  the  black  box  at  direct  support 
.  Cost  of  the  bits  and  pieces  needed  at  direct  support 

•  Manpower  needed  to  provide  the  bits  and  peices  at  direct  support 
.  Cost  of  loaded  manpower  for  various  categories  of  direct  labor 

.  Average  lifetime  of  a  black  box 

•  Cost  of  various  categories  of  loaded  manpower 

Sources  of  data  include  published  reports  (and  unpublished  back-up  material), 
equipment  catalogues,  and  financial  summaries.  Typical  sources  of  data  for  the 
example  given  previously  are  indicated  below. 


The  number  of  complaints  per  month,  N 


The  quantity  N  is  composed  of  usage  rate,  reliability,  and  the  number  of 
black  boxes  in  use.  If  n  black  boxes  are  being  used,  each  an  average  of  t  hours  a 
month,  and  if  the  complaint  reliability,  l.e.,  the  mean  time  between  complaints 
(MTBC)  on  the  black  box,  is  ac  complaints  per  hour  of  use,  then 


The  MTBC  can  be  estimated  from  field- failure  data. 


The  usage  rate,  t,  must  be  estimated  from  deployment  plans,  as  must  the 
number,  n,  of  boxes  in  use.  Ch  a  projected  system,  all  these  factors  will  be 
available.  Hence  N  can  be  estimated. 

The  fraction  of  verified  complaints 

The  fraction  of  verified  complaints,  v,  is  another  output  of  the  observation 
of  field  failure  data.  If  Ay  is  the  mean  time  between  verified  complaints,  then, 

V  =  T$- 

V 

The  fraction  of  verified  complaints  that  can  be  disposed  of  at 
the  helicopter 

The  fraction  of  verified  complaints  that  can  be  disposed  of  at  the 
helicopter,  f,  is  estimated  essentially  from  two  pieces  of  information: 

(1)  A  maintenance  plan  that  defines  the  repairs  that  will  be  made  at  the 
helicopter.  (Adjustments,  for  instance,  will  often  be  made  there.) 

(2)  A  reliability  prediction  in  greater  detail  than  those  needed  for  Ac  and 
~hf  ,  namely,  a  breakdown  of  A  into  those  cases  which  will  be  disposed 
of  at  the  helicopter  and  those  which  will  have  to  go  to  direct  support. 
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Such  predictions  can  be  obtained  ircm  a  more  detailed  knowledge  of  the 
equipment;  or,  alternately,  statistical  values  of  the  fraction  /  for 
similar  equipments  now  in  service  can  be  used  as  estimates. 

The  cost  of  maintenance  manpower 

Direct-maintenance  manpower  is  obtainable  from  maintainability  predictions. 
Suppose  the  direct-maintenance  time  required  and  the  corresponding  hourly  pay 
required  on  the  average  to  accomplish  the  maintenance  actions  are,  respectively, 

and  C1 

where  the  subscript  i  refers  to  the  skill  class.  Then  can  be  obtained  from 
maintainability  predictions;  can  be  obtained  Iran  lists  of  pay  classes,  together 
with  an  estimate  of  the  useful  life  of  maintenance  men  in  grade.  Here  a  suitable 
definition  of  useful  life  might  be  the  percentage  of  the  time  in  grade  during 
which  the  man  is  actually  assigned  to  maintenance  duties. 

Besides  direct  labor,  there  is  in  any  organization  a  great  deal  of  overhead 
labor.  Much  of  this  is  concerned  with  scheduling  and  supervision,  and  some  with 
management;  and  a  good  deal  goes  to  leave,  training,  and  normaintenance  duties 
of  the  men  themselves. 

In  general,  the  loaded  time  will  be  a  linear  function  of  the  direct-labor 
time.  If  is  the  loaded  time  (direct  and  overhead)  spent  in  labor  class  i,  then 


The  cost  of  labor  is  then  given  by 

C(T)  -£  v,  -  I  aici  +  l  Wi 

The  constants  involved  in  the  equation  above  are  the  overhead  coefficients, 
and  b^.  Rough  estimates  of  these  can  be  made  firm  tables  of  organizations 
and  fan  estimated  work-loads. 

A  good  approximation  to  the  equation  above  can  be  obtained  in  the  form 

C (T)  "  Ca  +Cb  .t 

where  t  is  the  total  active- repair  time  in  all  labor  classes. 

The  materials  costs  at  supply 

The  materials  costs  at  supply  ‘are  the  parameters  that  connect  the  different 
echelons.  To  estimate  them,  two  kinds  of  information  are  needed:  (l)the  average 
amount  and  kinds  of  materials  needed  to  perform  repairs,  and  (2)  the  cost  of  these 
materials  to  supply.  These  categories  are  discussed  below. 
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Estimates  of  the  kinds  and  quantities  of  materials  can  be  obtained  either 
tfcm  a  statistical  analysis  of  the  behavior  of  similar  equipments  in  present  use 
or  from  a  detailed  reliability  analysis,  based  on  actual  schematics. 

The  cost  of  these  materials  at  the  supply  echelon  in  question  will  consist 
of  the  following: 

•  Cost  at  the  higher  supply  echelon,  which  supplies  the  one  in  question 

•  Cost  of  the  labor  needed  at  the  higher  echelon  and  at  the  echelon  in 
question  to  move  the  materials 

•  Cost  of  transportation 

The  materials  cost  at  one  echelon  then  contains  implicitly  the  accrued  supply 
costs  at  all  higher  echelons;  thus  the  whole  system- support  cost  will  accounted 
for. 


The  cost  of  supply  manpower 

The  labor  cost  at  supply  must  be  obtained  by  an  analysis  similar  to  that 
described  for  the  maintenance  manpower.  If  a  detailed  analysis  is  not  available, 
probably  the  best  estimate  obtainable  is  to  assume  that  every  action,  i.e.,  every 
requisition  and  every  issue,  takes,  on  the  average,  about  as  much  labor  as  every 
other.  Then  if  the  total  payroll  of  the  supply  organization  is  divided  by  the 
number  of  pieces  of  paper  generated,  an  estimate  of  the  labor  cost  of  requisitions 
and  issue  is  obtained. 

The  time  dclav:  the  average  life  of  a  black  box 

The  cost  of  time  delay  [involving  AT,  L.  and  CQ  --  the  time  delay,  the 
average  life  of  a  black  box  in  the  (partial)  system,  and  the  cost  of  the  black 
box  to  supply,  respectively]  has  been  given  as 


The  estimation  of  C  lias  been  discussed  above, 
c 

observations  on  similar  support  organizations, 
from  condemnation  rates,  return  rates  to  higher 
total  number  in  circulation. 

If  on  an  average  a  black  boxes  a  month  are 
higher  echelons  of  maintenance  for  repair,  then 
tion, 

^  "  a  +  b 

Again  a  and  b  can  be  estimated  from  a  detailed  reliability  analysis  and  a 
maintenance  plan,  or  from  statistical  values  for  similar  equipment.  The  number 
n  is  determined  by  the  verified  failure  rate  and  by  logistic  policy. 


Time  delays  must  be  estimated  by 
The  average  life  can  be  estimated 
echelons  of  maintenance,  and  the 

condemned,  and  b  are  returned  to 
if  there  are  n  boxes  in  circula- 
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2.12  EXERCISE  THE  MODEL 

2.12.1  General 

The  method  used  in  exercising  the  model  is  dependent  on  the  type  and  complex¬ 
ity  of  model  and  the  time,  equipment,  personnel,  and  money  available. 

For  a  complex  model,  the  time  and  cost  involved  in  exercising  the  model  on 
a  computer  may  be  much  less  than  the  time  and  cost  for  using  people  with  slide 
rules.  However,  if  this  model  is  only  required  t  o  be  exercised  once,  the  cost 
of  set-up  time  for  the  computer  may  preclude  its  use.*  In  general,  the  computer 
is  much  faster  for  exercising  a  model  than  using  a  slide  rule  or  a  desk  calculator 
once  the  model  has  been  programmed.  In  actual  practice,  however,  the  analyst  has 
an  overall  time  constraint  on  the  study  effort,  and  if* a  computer  is  available, 
the  time  to  exercise  the  model  will  not  change,  because  the  analyst  will  use  any 
additional  time  to  conduct  sensitivity  analyses  or  expand  the  original  model. 

The  basic  point  to  be  made  in  exercising  the  model  is  that  no  matter  what 
process  is  used  —  desk  calculator,  computer,  or  slide  rule  —  the  final  result 
will  be  only  as  good  as  the  model  and  the  information  put  into  the  model.  The 
use  of  a  computer  does  not  in  itself  ensure  a  more  valid  result. 

2.12.2  Analysis  of  Output  Data 

When  an  analytical  model  has  been  developed  and  sufficient  input  data  gathered, 
the  model  can  be  exercised,  either  manually  or  by  means  of  computer.  In  the  sim¬ 
plest  of  cases,  a  single  dependent  variable  will  result  from  the  process.  In  most 
systems  analyses,  however,  a  whole  family  of  dependent  variables  will  be  generated. 

2.12.2.1  Analysis  of  a  Single  Dependent  Variable 

Each  model  equation  will  yield  one  output  parameter  (the  dependent  variable) 
when  one  set  of  input  parameters  (independent  variables)  is  used.  The  output 
parameter  might  represent  an  average,  predicted,  or  estimated  value.  The  value 
could  represent  a  measure  of  cost,  effectiveness,  reliability  or  any  other  param¬ 
eter  of  interest  upon  which  the  model  was  based. 

The  single  output  parameter  could  of  course  be  analyzed  by  comparing  it  to 
some  previously  known  standard  of  acceptability.  For  example  the  objective  of  the 
analysis  may  have  been  to  estimate  the  reliability  of  a  product  to  determine  com¬ 
pliance  with  a  pre-established  requirement.  In  this  instance,  a  judgment  of 
acceptability  of  the  product  might  be  made  by  simply  comparing  the  estimated 
value  with  the  required  value. 

In  many  cases,  however,  it  is  desirable  (and  often  necessary)  to  analyze 
the  resultant  from  the  standpoint  of  the  associated  uncertainties. 

•k 

Tins  is  one  area  in  which  a  cost-effectiveness  study  could  be  used  to  deter¬ 
mine  whether  a  computer  should  be  employed. 
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While  often  difficult  to  admit,  various  reasons  contribute  to  uncertainties 
which  affect  the  nature  of  estimates,  measurements,  or  expected  outcomes.  These 
uncertainties  are  not  necessarily  indicative  of  faulty  or  careless  estimation 
procedures,  but  rather  are  reflective  of  the  difficulty  in  characterizing  the 
real  world.  From  previous  experience  ample  evidence  exists  that  the  process  of 
prediction  is  a  difficult  one.  History  tells  us  that  the  estimated  cost  of  build¬ 
ing  the  Suez  Canal  was  off  by  a  factor  of  twenty.  Cost  estimates  for  recently 
built  nuclear  power  plants  have  been  half  of  what  they  eventually  wound  up  cost¬ 
ing. 

In  9  out  of  22  major  hardware  systems  studied  recently  by  the  Rand  Corpora¬ 
tion,  the  actual  procurement  cost  per  article  was  more  than  three  times  an  early 
estimate  of  cost;  and  in  two  cases  the  actual  cost  was  more  than  ten  times  an 
early  estimate. 

The  problem  of  uncertainty  in  estimating  cost  and  effectiveness  for  new  sys¬ 
tems  can,  of  course,  never  be  eliminated.  However,  the  analyst  can  minimize  un¬ 
certainty  and  --  more  important  --  can  account  for  it  in  providing  information 
to  the  decision-maker.  This,  of  course,  requires  knowledge  of  the  types  of 
uncertainty  which  might  be  encountered. 

Charles  Hitch  and  Roland  McKean,  in  their  book, The  Economics  of  Defense  in 
the  Nuclear  Age,  (Harvard  University  Press,  19&3)  describe  five  basic  types  of 
uncertainty  associated  with  estimates: 

Uncertainty  about  planning  and  cost  factors 

(b)  Uncertainty  about  strategic  context 

(c)  Technological  uncertainty 

(d)  Uncertainty  about  the  enemy  and  his  reactions 

(e)  Statistical  uncertainty 

The  analyst  is  confronted  with  the  task  of  deciding  how  uncertainties  are  to 
be  treated.  The  most  important  advice  is  of  course,  "Don't,  ignore  them".  Secondly 
he  must  be  able  to  recognize  the  type  of  uncertainty  involved.  Third,  he  must  be 
able  to  distinguish  between  the  important  and  unimportant  uncertainties  in  con¬ 
text  with  the  particular  analysis.  Finally,  he  must  be  able  to  expand  on  his 
basic  estimate  or  measurement  by  additional  consideration  of  the  contingencies 
created  by  the  uncertainty.  This  may  entail: 

(a)  Expressing  the  dependent  variable  as  a  range  of  values,  each  value 
having  a  probability  of  occurrence. 

(P)  Assigning  confidence  intervals  about  the  estimate. 

(C)  Subjectively  qualifying  the  nature  of  the  estimated  or  measured 
value. 

Figure  2.9  illustrates  six  different  ways  of  expressing  an  estimated  value. 
Each  successive  expression  form  represents  a  higher  degree  of  specificity  in 
treating  uncertainty. 
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EXPRESSION  FORM 

DEGREE  CF  SPECIFICITY 

1.  System  A  is  estimated  to  cost  $15M. 

1.  Nj  uncertainty  expression. 

2.  System  A  is  estimated  to  cost  $15M; 
however,  the  analyst  is  not  sure 
(uncertain)  about’ the  figure. 

2.  A  vague  qualitative  expression 
of  uncertainty  is  given. 

3.  System  A  is  estimated  to  cost 
between  $11M  and  $19M. 

3.  A  range  is  given  to  express  the 
magnitude  of  uncertainty.  Fbvv- 
ever,  no  probability  information 
is  given;  it  is  not  stated 
whether  the  analyst  believes 
there  is  a  1 %,  a  10$  or  a  100$ 
chance  that  the  cost  will  fall 
between  $11M  and  $19M,  nor  i s 
it  indicated  whether  the  cost 
is  likely  to  be  closer  to  $11M 
or  $19M. 

4.  There  is  a  "strong  probability" 
that  System  A's  cost  will  be: 

$11M  -’$15M  -  $19M.  The  $15M  is 
some  measure  of  central  tendency 
(mean,  mode  or  mean).  The  $11M’ 
and  $19M  are  the  estimated  lower 
and  upper  cost  limits. 

4.  An  adjective  descriptor  is  added 
to  convey  a  rough  indication  of 
probability  . 

5.  With  a  .95  probability.  System 

A's  cost  is  estimated:  $IlM  - 
$15M  -  $19M.  The  numerical 
expressions  have  the  same  meaning 
as  in  4  above. 

5k  The  adjective  descriptor  is 

replaced  by  the  more  definitive 
numeral. 

Probability 

cost 

6.  A  complete  probability  distribu¬ 
tion  is  given,  and  this  is 
depicted  by  a  curve.  (Both  the 
problems  in  getting  the  Case  6 
type  information  and  the  amount 
of  additional  information  pro¬ 
vided  by  Case  6  are  of  a  greater 
magnitude  vs  Case  5  than  Case  5 
is  vs  Case  4,  Case  4  is  vs 

Case  3,  etc.) 

FIGURE  2-9 

EXPRESSIONS  OF  UNCERTAINTY 


2.12.2.2  Analysis  of  Several  Dependent  Variables 

Most  systems  analysis  problems  encountered  will  involve  the  treatment  of 
more  than  one  dependent  variable.  For  example,  quite  often  two  alternative 
systems  or  alternative  designs  are  the  subject  of  the  analysis.  Further,  at 
least  two  dependent  parameters,  —  e.g.,  cost  and  effectiveness  —  (and  probably 
many  more)  are  of  interest  in  making  the  comparison.  An  even  more  complex  sit¬ 
uation  arises  when  the  study  objectives  involve  "trade-offs"  where  the  dependent 
variables  of  interest  can  assume  a  broad  range  of  values.  In  general,  the  task 
of  analyzing  data  outputs  can  be  subdivided  into: 

*  Comparative  Analyses 
'General  Trade-off  Studies 
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Comparative  analyses,  as  defined  herein,  are  studies  aimed  at  determining 
which  of  two  or  more  alternatives  is  better  (or  best).  If  a  single  dependent 
variable  constitutes  the  basis  for  comparison,  the  analysis  is  a  rather  simple 
one.  Hence,  the  problem  might  be: 

*  Which  of  two  systems  is  more  effective?  or 

’  Which  of  two  systems  is  cheaper?  or 

’  Which  of  two  systems  is  cheaper  and  better? 

In  treating  the  above  types  of  problems,  the  analyst's  responsibility  is  to 
recommend  a  single  choice  from  among  the  alternatives.  Little  is  left  for  the 
decision-maker,  except  to  accept  or  reject  the  results  of  the  analysis. 

In  many  cases,  however,  the  analyst's  objective  is  to  present  more  than  one 
alternative  to  the  decision-maker.  For  example,  the  terms  of  the  analysis  may 
require  that  separate  values  of  cost  and  effectiveness  be  presented  for  each 
system  considered,  with  final  choice  of  the  "better  one"  being  the  decision¬ 
maker's  choice.  An  example  of  this  approach,  given  below,  shows  that  system  B 
costs  more  that  system  A  —  but  is  also  more  effective.  In  this  example,  the 
judgment  of  the  relative  worth  of  the  two  systems  could,  by  intent,  be  that  of 
the  decision-maker. 

System  cost  Effectiveness 

A  $1 . OM  .95 

B  $1.5M  ,98 

A  similar  example  is  one  wherein  the  decision-maker  desires  to  subjectively 
consider  certain  factors  which,  by  intent,  have  not  been  included  in  the  model. 
(These  factors  are  called  "leverage  effects"  and  will  be  discussed  in  greater 
detail  in  Section  2.13.)  An  example  of  such  a  case  is  evident  in  the  analytical 
process  recently  employed  to  select  a  Chief  of  Police  for  the  City  of  Los  Angeles 
In  that  situation,  there  were  several  candidates  for  the  position.  By  means  of 
a  vigorous  analytical  approach  (actually  a  model  was  developed  with  which  to 
estimate  the  "effectiveness"  of  each  candidate),  the  number  of  choices  was  re¬ 
duced  to  three.  The  ultimate  selection  of  the  best  man  for  the  job  was  made 
by  choice  of  the  decision-maker  (in  this  case,  the  top  City  officials).  In  the 
example  cited,  it  is  to  be  noted  that  the  analytical  model  was  used  to  minimize 
the  number  of  possible  alternatives,  but  was  not  necessarily  used  to  arrive  at 
the  final  decision. 

Often,  the  objective  of  the  analyst  is  to  conduct  trade-off  studies  where 
two  or  more  dependent  variables  are  to  be  considered  over  a  broad  range  of 
possibilities.  This  form  of  objective  is  quite  common  during  the  concept  formu¬ 
lation  phase  of  a  project  when  requirements  are  being  developed.  jn  thjs  situa¬ 
tion,  the  analyst  has  been  given  a  minimum  level  of  acceptability  and  a  design 
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goal  for  the  dependent  variables  of  interest,  ft  then  must  analyze  the  man} 
alternatives  within  the  given  envelope  in  order  to  reduce  the  number  of  choices 
to  one  —  or  to  some  minimum  number.  Trade-offs  can  involve  weighing  one  per¬ 
formance  parameter  against  another  or  weighing  performance  against  cost.  Examples 
of  each  situation  are  given  below: 

(1)  Performance  Parameter  Trade-offs 

One  of  the  major  parameters  in  most  effectiveness  models  is  avail¬ 
ability.  Availability  is  in  turn  a  function  of  reliability  and  maintain¬ 
ability.  When  reliability  is  expressed  as  a  frequency  of  failure  (MTBF) 
and  maintainability  expressed  as  the  length  of  time  required  to  restore 
(MDT)  a  failed  item,  availability  in  some  cases  can  be  expressed  as 

MTS7 

A  =  MTBF  +  MM 

If  a  fixed  level  of  availability  is  the  desired  output,  it  is  apparent 
that  MIBF  and  M3T  can  be  traded  off  in  achieving  the  desired  value. 

(2)  Performance  Versus  Cost  Trade-offs 

Another  form  of  trade-off  problem  involves  performance  versus  cost. 

In  this  type  of  trade-off  problem  the  principle  objective  is  to  weigh 
varying  levels  of  performance  against  varying  levels  of  cost.  Gctnmcn 
trade-offs  in  this  category  include: 

‘  Speed  versus  cost 
‘  Payload  versus  cost 
'  Reliability  versus  cost 

Considerable  emphasis  has  been  directed  recently  to  the  consideration 
of  total  life- cycle  costs  when  formulating  system  level  decisions.  The 
reasoning  behind  the  emphasis  is  that  incomplete  consideration  of  the 
influencing  factors  often  can  lead  to  erroneous  decision.  A  case  in 
point  can  be  illustrated  by  the  following  example. 

It  is  assumed  that  the  decision-maker  must  choose  between  two  systems 
of  differing  availability  on  the  basis  of  cost.  It  may  be 
generally  shown  that  development  cost  and  initial  investment 
cost  increase  with  increased  reliability  (see  Figure  2.10a). 

Ch  the  other  hand  annual  operating  costs  (and  hence  total  recur¬ 
ring  costs)  decrease  with  increased  reliability  (see  Figure 
2.10b).  It  readily  becomes  apparent  that  the  cheaper  system 
can  only  be  determined  by  combined  consideration  of  all 
costs  (Figure  2,10c). 
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(a) 


(b) 


(c) 


FIGURE  2-10 
TRADE-OFF  CURVES 


(3)  Trade-off  Analyses  Related  to  Project  Objectives 

Trade-off  analyses  are,  in  general,  meaningless  unless  placed  in  con¬ 
text  with  a  specific  set  of  objectives.  For  example,  during  concept  formu¬ 
lation,  the  following  objectives  are  significant: 

(a)  Establish  requirements  (or  envelopes). 

Select  a  minimum  set  of  design  alternatives. 

Prove  the  technical,  economic,  and  military  feasibility 
of  selected  design  alternatives. 
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The  interdependencies  between  trade-off  studies  and  the  above  objectives 
are  shown  in  Figure  2.11.  Any  trade-off  study  must  be  based  on  a  set  of 
dependent  and  independent  variables.  The  status  of  a  given  project  in  terms 
of  accomplishment  of  the  objectives  would  identify  the  dependent  and  inde¬ 
pendent  variables  for  the  trade-off  study  effort.  For  example,  if  require¬ 
ments  envelopes  have  been  established,  these  constitute  the  set  of  indepen¬ 
dent  variables  for  the  trade-off  study.  The  objective  of  the  study  might 
then  be  the  selection  of  a  minimum  set  of  design  alternatives. 


FIGURE  2-11 

INTERDEPENDENCIES  BETWEEN  TRADEOFF 
STUDIES  AND  PROJECT  OBJECTIVES 
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(4)  Sensitivity  Analysis 

An  important  tool  in  performing  trade-off  studies  is  the  analysis  of 
sensitivity.  In  general,  sensitivity  analysis  involves  determining  the 
significance  of  a  given  variable  within  some  prescribed  range  of  interest. 
A  simple  illustration  of  sensitivity  is  given  in  Figure  2-12,  which  des¬ 
cribes  the  relationship  between  total  system  cost  and  the  endurance  for 
a  hypothetical  manned  aircraft.  The  analyst  would  conclude  from  the  curve 
that: 

(a)  Cost  is  highly  sensitive  to  endurance  at  low  levels  of  endurance. 

(b)  Cost  is  relatively  insensitive  to  endurance  of  higher  values  of 
endurance. 


FIGURE  2-12 
SENSITIVITY  ANALYSIS 


2.13  DEVELOP  DECISION  MODEL 

2.13.1  Optimization  Criterion 

In  defining  an  optimizing  criterion,  the  system  analyst  is  faced  with  a 
problem  similar  to  that  of  putting  in  precise,  quantifiable  terms  the  rules 
or  criteria  for  choosing  the  "best"  painting  or  "best"  automobile.  These  ex¬ 
amples  do  have  quantifiable  characteristics,  such  as  the  size  of  the  painting 
or  cost  of  the  automobile;  however,  artistic  judgment  and  user  experience, 
respectively,  are  factors  in  the  final  choice.  In  the  same  sense,  the  choice 
of  the  best  system  is  greatly  influenced  by  the  use  of  good  engineering,  eco¬ 
nomic.  and  operational  judgment. 
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It  is  most  important,  however,  that  the  optimizing  criterion  be  defined  to 
the  maximum  extent  possible,  for  the  following  reasons: 

The  inputs  provided  to  the  analyst  through  use  of  the  criterion 
can  reduce  the  size  of  the  problem  to  a  point  where  a  judicious 
choice  can  be  made. 

Defining  a  criterion  forces  the  analyst  to  examine  all  possible 
alternatives  in  an  objective  manner  so  that  the  criterion  can  be 
adapted  to  mathematical  representation  and  analysis. 

(c)  It  is  easier  to  incorporate  the  ideas  and  experience  of  others 
if  a  formal  basis  for  optimization  is  established. 

(d)  The  (partial)  basis  for  final  choice  is  in  precise,  quantifiable 
terms  and  can  therefore  be  reviewed  and  revised,  and  can  provide 
inputs  to  a  learning-process  for  future  optimization  problems. 

When  a  criterion  for  optimization  is  being  formulated,  the  system  and  the 
boundaries  must  be  explicitly  defined.  This  definition  will  influence  the 
choice  of  parameters  in  the  optimization  model.  The  purchaser  of  a  new  auto¬ 
mobile,  for  example,  may  or  may  not  consider  the  service  policies  of  the  manu¬ 
facturer  and  dealer.  If  he  does,  the  system  is  both  the  automobile  and  service 
policies;  if  he  does  not,  the  -system  is  only  the  automobile.  In  attempting  to 
optimize  a  weapon  system  such  as  a  bomber,  the  analyst  has  to  consider  whether 
the  system  is  to  be  defined  as  a  single  bomber,  a  squadron  of  bombers,  or  the 
complete  bomber  fleet.  It  is  possible  that  optimizing  with  respect  to  a  single 
plane  (a  sub-optimization)  may  not.  yield  the  optimum  "squadron"  system. 

As  part  of  the  system-definition  process,  the  analyst  also  determines  the 
fixed  and  variable  factors  pertinent  to  the  system.  This  task  requires  a  pre¬ 
liminary  analysis,  since  consideration  of  all  possible  alternatives  will  usually 
lead  to  problems  of  unmanageable  size.  Some  factors  may  be  considered  fixed  if 
results  of  previous  analyses,  perhaps  sub-optimizations,  indicate  the  values 
that  have  attained  the  best  results  in  the  past.  The  maintenance  troubleshoot¬ 
ing  routine,  for  example,  might  normally  be  considered  as  a  variable  factor, 
but  past  research  in  this  area  may  be  used  to  select  a  particular  routine  appli¬ 
cable  to  the  system  under  study,  or  perhaps  to  restrict  the  range  to  several 
alternatives. 

Once  the  mission  profile  is  defined,  consideration  can  be  given  to  the 
physical  and  economic  limitations  that  will  have  to  be  imposed.  These  limita¬ 
tions  are  based  on  requirements  and  availabilities,  and  may  involve  such  factors 
as  minimum  system  output,  maximum  reliability,  maximum  development  time,  maximum 
weight  and  volume,  and  type  and  number  of  support  and  operational  personnel. 
Through  such  consideration  and  envelope  of  design,  development,  operational, 
and  support  alternatives  can  be  established  in  such  a  way  that  each  overall 
configuration  within  the  envelope  will  meet  physical  and  economic  limitations 
as  well  as  minimum  performance  goals. 
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Nw  the  analyst  must  select  a  decision  criterion  by  specifying  the  types  of 
effectiveness  and  cost  parameters  to  be  investigated  and  by  assigning  numerical 
values  where  required.  The  choice  of  objectives  and  criteria  is  perhaps  the 
most  difficult  task  in  system  effectiveness  optimization.  It  is  expected,  how¬ 
ever,  that  current  research  in  the  optimizing  of  system  effectiveness  will 
develop  theory  and  accumulate  experience  to  help  overcome  some  of  the  difficul¬ 
ties  of  this  task. 

It  would  be  impossible  to  establish  rigid  ground  rules  or  procedures  for 
formulating  a  criterion  for  optimizing  system  effectiveness.  The  answers  to 
the  following  two  basic  questions,  however,  will  provide  a  great  deal  of  in¬ 
sight  for  such  formulation: 

(1)  Wy  is  the  system  being  developed? 

(2)  What  physical  and  economic  limitations  exist? 

The  answer  to  the  first  question  essentially  defines  the  mission  profile  of 
the  system.  Where  possible,  the  definition  should  be  translated  into  quantita¬ 
tive  parameters  --  a  difficult  task  in  many  cases.  A  performance  measure  such 
as  kill- probability  for  a  SAC  bomber  may  be  assignable,  but  the  bomber  may  also 
have  a  mission  to  act  as  a  deterrent  --  a  measure  that  is  difficult,  if  not 
impossible,  to  quantify.  It  is  for  this  type  of  multi-mission  case  that  judg¬ 
ment  will  become  especially  important.  Even  if  quantitative  requirements  can 
be  placed  on  all  mission  types,  weighting  factors  would  have  to  be  introduced 
to  quantify  the  relative  importance  of  each  mission. 

Factors  that  have  relatively  little  impact  on  overall  effectiveness  or  cost 
can  be  considered  to  be  fixed  or,  possibly,  can  be  ignored.  There  is,  of  course, 
a  risk  involved  if  factors  chosen  to  be  fixed  or  unimportant  would  have  had  a 
significant  effect  if  they  had  been  allowed  to  vary.  Factors  that  fall  in  this 
"gray  area"  may  have  constraints  imposed  upon  them  in  such  a  maimer  that  the 
more  detailed  analysis  to  be  performed  in  the  optimization  process  will  indi¬ 
cate  final  disposition.  For  example,  if  a  questionable  factor  might  have  a 
monotonic  influence  on  effectiveness,  consideration  of  only  extreme  values 
might  be  all  that  is  necessary  to  determine  the  significance  of  this  influence. 

It  is  important  that  factor  selection,  variability,  and  the  final  choice  of 
system  definition  be  clearly  indicated  so  that  the  scope  of  the  optimization 
process  will  be  known  and  areas  for  possible  modification  of  the  formal  mathe¬ 
matical  solution  will  be  made  explicit. 

2.13.2  Risk  and  Uncertainty 

It  is  rare  for  a  decision  not  to  include  some  degree  of  risk  and  uncertainty. 
In  many  cases,  the  risks  can  be  identified  before  the  decision  is  made,  and  their 
effects  can  be  included  in  the  analysis.  Some  degree  of  control  is  thus  obtained 
over  risks  and  uncertainties,  making  it  possible,  for  instance,  t0  specif}  how 
much  risk  can  be  tolerated. 
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A  distinction  should  be  made  between  risks  and  uncertainties.  A  probability 
can  be  assigned  to  any  event  that  is  considered  a  risk,  but  no  probability  can 
be  assigned  to  an  uncertainty.  An  example  of  a  risk  is  a  gambler  making  a  bet 
that  he  will  draw  a  red  ball  from  an  urn  containing  5  red  balls  and  10  white 
balls.  The  possible  outcomes  are  known,,  and  the  probabilities  are  1/3  for  a 
red  ball  and  2/3  for  a  white  ball.  An  example  of  uncertainty  is  making  the 
same  bet  where  the  number  of  red  balls  and  the  number  of  white  balls  are  unknown. 
In  this  case,  all  that  can  be  said  about  the  outcome  is  that  a  red  ball  or  a 
white  ball  will  be  drawn. 

In  practice,  the  distinction  is  not  always  clear.  It  may  be  known,  for 
example,  that  the  number  of  red  balls  is  between  five  and  ten  and  the  number  of 
white  balls  equal  ten.  Since  analyses  under  conditions  of  risk  are  preferred 
to  those  under  conditions  of  uncertainty,  some  effort  must  be  made  to  learn  more 
about  the  system  and  thereby  reduce  the  amount  of  uncertainty  in  the  decision. 

2 . 13 . 3  Optimization  Techniques 

The  technique  for  optimization  essentially  involves  the  application  of 
effectiveness  and  cost  models  to  all  feasible  designs  and  selection  of  the  de¬ 
sign  which,  according  to  the  criterion,  is  optimum. 

While  this  approach  is  conceptually  simple,  its  implementation  is  virtually 
impossible,  except  for  the  most  simple  problems.  Consider  a  problem  involving 
fifteen  variables,  each  of  which  may  take  one  of  only  two  possible  values.  More 
than  32,000  possible  system  designs  would  have  to  be  considered,  a  magnitude 
that  would  tax  even  the  largest  of  the  available  computers. 

Techniques  are  therefore  needed  to  reduce  the  amount  of  mathematics  and 
computation  to  a  size  reasonable  for  computer,  geometrical,  or  even  hand  solu¬ 
tion.  In  a  sense,  these  techniques  are  sophisticated  trial- and- error  routines. 
Some  of  the  more  commonly  used  techniques,  or  fields  from  which  such  techniques 
are  derived,  are  listed  in  Table  2-3.  The  list  is  by  no  means  complete.  A 
brief  description  of  several  of  these  techniques  is  contained  in  Section  3  of 
the  Guidebook. 
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TABLE  2-3 

PARTIAL  LIST  OF  TECHNIQUES  FOR  OPTIMIZATION 

I. 

Mathematical  Techniques 

Birth  and  death  processes 

Calculus  of  finite  differences 

Calculus  of  variations 

Gradient  theory 

Numerical  approximation  methods 

Svmbolic  logic 

Theory  of  linear  integrals 

Theory  of  maximum  and  minimum 

11. 

Statistical  Techniques 

Bayesian  analysis 

Decision  theory 

Experimental  design 

Information  theory 

Method  of  steepest  ascent 

Stochastic  processes 

111. 

Programming  Techniques 

Dynamic  programming 

Linear  programming 

Nonlinear  programming 

IV. 

Other  Operations  Research  Techniques 

Gaming  theory 

Monte  Carlo  techniques 

Queuing  theory 

Renewal  theory 

Search  theory' 

Signal  flow  graphs 

Simulation 

Value  theory 

2.13.4  Leverage  Effects 

During  the  analysis  of  system  cost  or  cost-effectiveness,  a  great  deal  of 
emphasis  is  necessarily  placed  on  the  three  basic  types  of  cost:  research  and 
development,  investment,  and  operating.  However,  costs  and  benefits  in  another 
category  are  often  overlooked  during  these  analyses.  They  are  overlooked  be¬ 
cause  they  do  not  increase  or  diminish  the  total  cost  and  effectiveness  of  the 
system  being  analyzed.  These  costs  and  benefits  are  called  leverage  effects 
in  that  they  come  into  play  when  an  alternative  associated  with  the  system 
being  analyzed  influences  (acts  as  a  lever  on)  the  cost  or  some  other  charac- 
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teristics  of  another  system.  Leverage  effects  include  those  allied  factors  or 
elements  which  are  external  to  the  system  being  studied  but  may  have  a  signifi¬ 
cant  if  not  overwhelming  impact  on  the  final  choice  among  system  alternatives. 

Leverage  effects  need  not  be  expressed  as  a  quantity  of  dollars,  time,  or 
other  units.  They  can  be  simply  facts  or  circumstances  that  should  be  con¬ 
sidered  in  the  decision.  In  terms  of  the  definition  of  cost-effectiveness 
analysis  given  earlier,  a  leverage  effect  is  presented  as  one  component  of  the 
array  of  characteristics  mentioned  therein. 

Leverage  effects  can  be  illustrated  by  a  simple  example.  A  military  agency 
is  to  select  for  development  one  of  three  alternative  powerplants.  A,  B,  and  C, 
for  use  in  a  new  helicopter.  It  is  assumed  that  the  total  system  cost  and 
effectiveness  have  been  estimated  for  each  alternative  as  follows: 

Powerplant  A  Powcrplant  B  Powerplant  C 

Total  System  Cost  $60,000,000  $70,000,000  $80,000,000 

Effectiveness  o.95  0.95  0.95 

It  is  apparent  that  A  is  the  best  choice  since  it  is  the  least  costly  and 
provides  the  same  effectiveness.  However,  Powerplant  C  may  have  certain  quali¬ 
ties  that  would  permit  its  effective  use  in  a  new  tank  that  is  being  developed. 
It  is  assumed  that  the  total  cost  of  a  new  powerplant  developed  for  the  tank 
alone  would  also  be  $80,000,000,  but  that  if  Powerplant  C  were  selected  and 
developed  for  the  helicopter,  its  total  cost  would  be  reduced  to  $50,000,000 
because  of  shared  development  costs.  This  saving  of  $30,000,000  in  the  develop¬ 
ment  of  the  tank  now  makes  C  appear  to  be  the  best  buy.  The  $30, 000,000  is  a 
leverage  effect,  since  the  powerplant  for  the  helicopter  will  still  cost 
$80,000,000,  but  its  development  w  ill  effect  a  $30,000,000  saving  in  the  allied 
tank  program. 

It  might  be  argued  that  leverage  effects  could  be  included  in  either  the 
cost  or  effectiveness  values  of  the  system.  However,  the  analyst  is  forced  to 
isolate  the  problem  and  define  a  system  associated  with  it  in  order  to  perform 
the  analysis  at  a  manageable  level.  Although  the  leverage  effects  are  known  to 
influence  the  decision,  many  factors  must  be  excluded  if  the  system  is  to  be 
represented  by  a  model  and  pertinent  information  is  to  be  extracted  from  that 
model.  Then,  after  the  model  is  applied  to  the  various  alternatives,  some  of 
the  excluded  factors,  e.g.,  leverage  effects,  are  reconsidered  for  the  final 
decision. 

2.13.5  Interpretation 

As  indicated  previously,  a  model  of  a  complex  process  is  usually  incomplete 
because  of  uncertainties,  non- quantitative  factors,  inadequate  data,  and  inade¬ 
quate  consideration  of  the  effects  of  the  process  on  systems  and  operations  at 
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higher  echelons.  In  such  cases  the  results  of  the  optimization  process  can  only 
indicate  the  best  system  within  the  simplifications,  assumptions,  restrictions, 
and  omissions  required  to  circumvent  the  voids. 

The  effects  of  these  circumventions  must  then  be  evaluated  through  some 
type  of  model  feedback  procedure  which,  on  the  basis  of  the  attained  results, 
may  reveal  some  critical  deficiencies  that  can  be  rectified. 

However,  even  the  most  modem  mathematical  techniques  and  computers  will 
yield  only  partial  analytic  solutions,  mainly  because  of  the  uncertainties. 

These  uncertainties  often  exist  in  the  overall  objective  and,  when  broader  con¬ 
texts  are  being  considered,  it  may  be  necessary  to  examine  alternative  objectives. 
W;  thus  have  the  enlarged  problem  of  first  selecting  the  optimum  objective  and 
the  associated  optimum  set  of  constraints. 

The  optimization  process,  therefore,  provides  the  framework  for  a  final  de¬ 
cision.  If  the  process  is  based  on  a  correct  formulation  of  the  problem  and 
application  of  a  reasonable  model,  the  decision  can  be  critically  evaluated  and 
suitably  modified.  However,  because  of  our  present  inability  to  employ  a  strict¬ 
ly  analytical  approach,  the  experience  and  judgment  of  management  inherit  respon- 
sibity  for  the  final  choice. 

2.13  .6  Conclusion 

Ideally,  the  criteria  for  a  decision  should  be  explicitly  stated  so  that 
there  is  no  doubt  concerning  the  acceptability  and  accuracy  of  a  decision.  In 
most  cases,  however,  explicitness  is  impossible  because  of  the  uncertainties  that 
prevail  and  because  of  limitations  of  available  methods.  For  instance,  a  state¬ 
ment  about  the  reliability  of  a  device  is  meaningless  unless  a  standard  of  meas¬ 
urement  is  given.  Even  if  this  standard  is  given,  the  statement  would  still  be 
meaningless  unless  the  methods  for  making  the  measurement  were  available.  Un¬ 
certainties  about  the  validity  of  the  standard  of  measurement  and  the  accuracy 
of  the  method,  and  about  whether  the  criteria  are  proper  in  the  first  place, 
further  complicate  the  establishment  of  decision  criteria. 

The  appropriateness  of  decision  criteria  for  military  systems  is  a  contro¬ 
versial  point.  The  number  of  offensive  weapons  for  a  fixed  cost,  the  number  of 
targets  destroyed  for  a  fixed  cost,  the  number  of  lives  saved  for  a  fixed  cost  — 
which  should  be  used?  Selecting  a  particular  criterion  for  lack  of  a  better  one 
can  create  serious  problems  if  it  is  the  wrong  one.  However,  making  decisions 
in  the  absence  of  criteria  can  also  have  harmful  consequences.  An  understanding 
of  the  complexity  of  criteria  is  essential. 

The  normal  criteria  are  those  which  result  in  maximum  effectiveness  for  a 
given  budget  or  a  specified  effectiveness  at  minimum  cost.  However,  the  absolute 
value  of  gain  or  cost  must  not  be  overlooked,  as  it  would  be  in  simply  maximizing 
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the  ratio  of  effectiveness  to  cost.  The  ratio  of  effectiveness  to  cost  is  not 
generally  an  adequate  criterion  for  making  a  choice  among  competing  systems. 

The  absence  of  a  standard  criterion  does  not  preclude  the  analysis  of  cost- 
effectiveness.  It  means  that  as  much  information  as  possible  on  the  system  must 
be  derived  for  consideration  by  the  decision-makers.  Although  the  information 
cannot  be  "wrapped  up"  into  a  single  valid  criterion,  it  can  be  displayed  in  a 
manner  that  facilitates  its  use  in  conjunction  with  the  decision-maker's  expert 
judgments.  This  requirement,  however,  places  an  additional  burden  on  the  ana¬ 
lyst,  since  he  must  maintain  a  great  deal  of  flexibility  in  his  modeling  to  make 
his  analysis  adaptable  to  changing  information  requirements. 

2.14  EVALUATE  RESULTS  AND  FEEDBACK  INFORMATION 

The  major  function  of  the  evaluation  and  feedback  information  process  is 
to  provide  constant  updating  of  previous  inputs  and  analysis  by  using  the  in¬ 
formation  gained  from  the  study  process  as  it  becomes  available. 

For  example,  it  may  become  apparent  after  exercising  the  effectiveness 
model  that  the  alternatives  considered  in  the  analysis  are  all  extremely  vul¬ 
nerable  to  enemy  action.  At  this  point  in  the  systems  analysis/cost-effective- 
ness  process,  the  analyst  should  re-examine  the  mission  profiles,  threats,  and 
hardware  characteristics  to  determine  which  factors  are  contributing  to  the  high 
vulnerability.  If  the  mission  profiles  arc  causing  the  high  vulnerability  to 
some  avoidable  tactic,  this  information  should  be  fed  back,  to  the  decision¬ 
making  level  if  necessary,  so  that  the  mission  profiles  can  be  checked  for 
possible  changes.  If  the  threat  appears  to  be  causing  the  high  vulnerability, 
the  solution  may  be  to  go  back  and  consider  a  new  alternative  system,  or  possibly 
change  the  performance  requirements  of  the  system.  Other  considerations  to  be 
included  in  the  evaluation  and  feedback  process  are  the  following: 

’  Ensure  that  all  assumptions  and  subjective  judgments  used  in  the  analysis 
are  identified.  The  major  assumptions  should  be  explicitly  stated  at 
the  beginning  of  the  study  effort  and,  if  feasible,  examined  at  the 
decision-making  level  to  determine  if  the  assumptions  are  valid. 

'  Ensure  that  all  the  uncertainties  that  occur  in  the  analysis  are  treated. 
The  uncertainties  of  future  threats,  environments,  and  performance 
characteristics  may  have  probabilities  and  confidence  levels  associated 
with  them,  and  these  should  be  explicitly  stated. 

'  Examine  the  output  at  every  stage  in  the  systems  analyst. s/cost-eff ective- 
ness  process  to  determine  if  the  result  appears  to  be  correct.  Results 
that  are  intuitively  unexpected  may  lead  to  a  determination  that  some 
factor  in  the  analysis  was  inadvertently  omitted. 
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*  Include  parametric  treatment  of  assumptions  and  variables  found  to  be 
sensitive.  If  the  results  of  a  study  are  sensitive  to  the  assumption 
that  a  system  w  i  1 1  be  in  operation  for  say  fifteen  years,  go  back  and 
recalculate  for  one,  two,  and  five  years  on  either  side  of  fifteen. 

2.15  OUTPUT  RESULTS 

Once  the  need  for,  and  the  absence  of  any  suitable  substitute  for,  the 
decision-making  function  is  recognized,  attention  can  be  directed  to  the  infor¬ 
mation  requirements  of  the  decision-maker.  Preparing  appropriate  information 
and  presenting  it  to  the  decision-maker  are  the  functions  of  the  analyst. 

Essentially,  analysis  involves  providing  the  best  possible  estimate  of  the 
effects  of  selecting  various  courses  of  action.  The  decision-maker  must  decide 
which  set  of  effects  he  is  most  willing  to  accept.  (He  must  also,  of  course, 
judge  the  validity  of  the  estimates  presented  to  him,) 

One  pitfall  that  must  be  avoided  in  the  systems  analysis/cost  effectiveness 
process  concerns  the  amount  of  detail  the  analyst  presents  to  the  decision-maker 
who  uses  the  results  of  the  analysis.  The  analyst  must  exercise  extreme  care 
to  be  sure  that  the  decision-maker  is  aware  of  any  relevant  factors  that  were 
not  considered  in  the  analysis  or  that  may  have  been  obscured  by  the  data- 
rcduction  process  or  by  the  analysis  itself.  In  short,  since  data  reduction  and 
analysis  involve  some  decisions,  the  analyst  must  be  careful  not  to  make  decisions 
that  more  properly  belong  in  the  jurisdiction  of  the  decision-maker. 

It  is  emphasized  that  the  systems  analysis/cost-effectiveness  process  does 
not  represent  a  decision;  it  is  a  process  that  concludes  by  presenting  to  the 
decision-maker,  in  a  useful  format,  information  and  data  that  are  essential  to 
his  making  a  proper  decision.  The  array  elements  shown  in  Table  2-4  for  each  of 
three  competing  systems  represent  a  set  of  data  considered  by  the  analyst  to  be 
important  to  the  decision-maker.  However,  if' the  analyst  attempts  one  additional 
step  —  developing  a  single  cost-effectiveness  index  from  some  or  all  of  these 
data  --  he  may'  have  made  some  decisions  that  should  have  been  left  to  management. 
If,  for  example,  the  analyst  decides  that  the  index  is,  in  essence,  effectiveness 
divided  by  the  product  of  cost  and  time,  but  that  primary-mission  effectiveness 
is  five  times  as  important  as  the  secondary-mission  effectiveness  (Condition  I), 
system  B  is  apparently  the  preferred  system.  This  would  be  indicated  through 
the  use  of  weighting  factors;  that  is.,1^  and  K2  represent  the  weighted  "impor¬ 
tance"  of  the  prime  and  secondary  missions.  In  the  example  just  given  (Condition 
I)  K1  =  5K2. 
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TABLE  2-4 

ARRAY  OF  COST-EFFECHVENESS  CHARACTERISTICS 


cost  (C) 


Initial 

Invest- 

atlng 


Effectiveness 


e  Secondary 
on  Mission 


0. go 

2M  0.95 


7M  0.70 


0.0306 


0.0210 


C-E 

(Condition  II 
(%  =  K2) 


0.0270 

0.0266 

0.0251 


Cost-Effectiveness  (C-E)  is  defined  for  this  example  by  the  equation: 

C-E  -  ^ 

(Kj  +  Kg)  C  T 

For  Condition  I,  =  5K2 
For  Condition  11.  K1  =  K2 

Therefore  for  Condition  1,  the  C-E  of  system  A  is  calculated  as  follows: 


C-E  = 


KlEp  +  K2ES 
(Kx  +  K2)  C  T 

5(-90)  +  1(-7Q) 

(5  +  1)  (14.8)  (2) 

5,2 

177.6 


=  . 0292 

And  the  C-E  of  system  A  for  Condition  II  is  calculated  as  follows: 

K, E  +  K0E 
C-E  =  1  P  2  s 

(%  +  K2)  C  T 


_  l(.90)  +  1(.70) 

(l  +  l)  (14.8)  (2) 
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_  1.6 

59-2 

=  . 0270 

Therefore,  if  the  analyst  decides  that  primary  mission  effectiveness  is 
five  times  as  important  as  the  secondary-mission  effectiveness  (Condition  I), 

B  is  the  preferred  system.  However,  if  the  analyst  decides  that  the  primary 
and  secondary  missions  are  equally  important  (Condition  11),  system  A  now 
appears  to  be  the  preferred  system. 

A  favorite  method  for  presenting  the  results  of  a  cost-effectiveness  study 
is  a  curve  that  is  plotted  using  cost  as  the  abscissa  and  effectiveness  as  the 
ordinate. 

Certain  conclusions  can  be  drawn  concerning  the  shape  of  this  type  of 
curve.  In  general,  low  slopes  (a  large  gain  in  effectiveness  for  a  small  in¬ 
crease  in  cost)  are  the  most  desirable.  Two  examples  are  shown  in  Figure  2-13. 
Equipment  A  costs  $3,500  for  an  effectiveness  of  95  percent.  However,  by  a 
small  increase  in  the  cost.  Equipment  A  can  be  given  an  effectiveness  of  99 
percent.  Conversely,  for  Equipment  B.  unless  there  is  a  need  for  an  effective¬ 
ness  greater  than  98  percent,  no  more  than  $3,000  should  be  spent  on  this  equip¬ 
ment. 


Effectiveness  Effectiveness 


FIGURE  2-13 

EXAMPLES  OF  COST-EFFECTIVENESS  CURVES 
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The  prime  function  of  the  system  analyst  is  to  provide  the  decision-maker 
with  as  much  organized,  relevant  information  as  possible.  However,  this  informa¬ 
tion  does  not  automatically  identify  the  preferred  alternative,  because  a  common 
value-measure  cannot  always  be  developed.  In  making  such  final  selection,  the 
decision-maker  must  also  account  for  the  limitations  in  the  quantified  analysis, 
such  as  data  inadequacies,  modeling  assumptions,  and  uncertainty.  The  final 
selection,  therefore,  must  be  based  primarily  on  the  decision-maker's  judgment, 
the  information  in  the  C-E  array  providing  support  for  such  judgment. 
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CHAPTER  3 

TECHNIQUES 


There  are  three  broad  classes  of  techniques  that  can  be  applied  in  systems- 
analysis  and  cost-effectiveness  studies.  The  first  class  consists  of  techniques 
that  are  used  to  represent  a  system's  behavior  as  a  function  of  time  usually 
in  a  statistical  sense.  This  description  is  generally  probabilistic  in  form 
and  thus  relies  heavily  jn  the  theories  of  probability  and  statistics.  Moreover, 
since  the  representations  are  often  complex  (both  mathematically  and  physically), 
a  computer  is  often  required  to  manipulate  or  solve  them. 

Within  this  first  class,  there  are  four  techniques  that  are  generally 
applicable  in  the  syatems-analysis/cost-effectiveness  process. 

•  Simulation  •  Sequencing 

•  Queuing  Theory  •  Inventory  and  Replacement 

The  second  class  of  techniques  applicable  to  systems  analysis/cost- 
effectiveness  is  concerned  with  finding  optimal  solutions,  i.e.,  the  maximization 
or  minimization  of  some  objective  function  within  specified  constraints.  Within 
this  class  are  the  following  techniques: 

•  Linear  and  Dynamic  Programming  •  Analytic  Models 

•  Game  Theory  •  Decision  Theory' 

•  Information  Theory 

The  third  class  of  techniques  consists  of  those  statistical  and  mathematical 
tools  used  by  the  analyst  to  Identify  relationships  among  such  system  parameters 
as  cost,  performance,  etc.,  and  determine  how  critical  the  parameters  are  in 
the  decision-making  process.  This  class  includes  the  following  techniques: 

•  Estimating  relationships  *  Experience  Curves 

•  Confidence  Intervals  •  Sensitivity  Analysis 

Table  3-1  lists  the  various  techniques  introduced  above,  indicates  the 
general  application  of  each  one,  and  identifies  the  section  in  this  guidebook 
in  which  the  technique  is  discussed. 
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TABLE  3  1 

SYSTEMS  ANALYSIS/COST  EFFECTIVENESS  TECHNIQUES 

Applications 

Technique 

Inter-  and 
Intra-Sy stem 
cost- 

Effectiveness 

Comparisons 

Selec¬ 

tion 

of 

Optimum 

Mixes 

System 
Effec- 
tivene  se 
Studies 

cost 

Studies 

Identifica  tior 
of  Critical 
Parameters 
and  System 
Problem"  Areas 

Loca¬ 
tion 
i  n 

Guldf 

Book 

Simulation 

X 

X 

X 

3.1 

Queuing  Theory 

X 

X 

3-2 

Sequencing  and 
Markov  Processes 

X 

X 

3.3 

Inventory  and 
Replacement 

X 

X 

X 

3.4 

Linear  and 

Dynamic  Programming; 

X 

X 

X 

3.5 

iGanc  Theory' 

X 

X 

X 

3.6 

Information  Theory' 

X 

X 

3.7 

.Analytic  Models 

X 

X 

X 

3.8 

•Decision  Theory 

X 

X 

3.9 

Cost -Estimating 
Relationships  and 
Confidence 

Intervals 

X 

X 

X 

: 3.io 

Experience  Curves 

X 

X 

33.  u 

Cost-  Sensitivity 
Analysis 

X 

X 

X 

;3.i2 

3.1  SIMULATION 

A  simulation  is  a  model  (usually  computer)  that  duplicates  a  system's  behavior 
without  actually  employing  the  system. 

A  simulation  can  be  employed  in  many  types  of  systems  analysis.  g^g  0f 
the  more  important  areas  and  circumstances  are: 

•  Environmental  problems 

•  Mathematical  formulation 

.  Lack  of  analytical  solution  technique 

•  Experimental  impossibility  --  e.g.,  large-scale  conflict 

•  cost 
.  Time 

•  Training 
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Simulations  can  be  either  analog  or  digital,  and  both  have  been  applied  to 
a  host  of  problems.  Within  the  set  of  simulations  are  several  other  concepts 
such  as  Monte  Carlo.  Gaming,  Training  Devices,  and  Model  Sampling. 

Analog  simulations  are  most  often  used  as  a  means  of  solving  sets  of 
differential  equations  or  problems  dealing  with  continuous  functions.  Generally, 
the  systems  analyst  is  more  likely  to  encounter  digital  simulations  in  the 
exercise  of  his  studies. 

The  question  might  properly  be  asked:  how  is  a  digital  simulation  of  a 
complex  system  obtained,  say,  for  a  forward- area  air- defense  problem?  The 
following  steps  are  necessary: 

(1)  The  characteristics  of  the  offense,  defense,  and  environment  are 
determined. 

(2)  A  general  flow  diagram  for  the  simulation  is  developed  —  for  example, 
the  flow  between  the  threat,  detection  and  tracking  radars,  and  the 
intercepting  missile. 

(3)  Detailed  flow  diagrams  and  submodels  are  developed  —  for  example, 
the  method  of  computing  the  lock-on  probability  for  the  tracking 
radar. 

(4)  Space  and  time  coordination  are  developed  throughout  the  simulation 
for  each  simulation  element. 

(5)  Statistical  sizes  and  constraints  arc  determined. 

(6)  Inputs  are  incorporated. 

(7)  The  simulation  model  is  exercised. 

Ar  important  aspect  of  Monte  Carlo  game  simulations  is  the  Design  of 
Experiments  for  testing  numerous  variables  and  reducing  output  variance  while 
reducing  the  required  sample  size.  A  formal  branch  of  statistics  is  devoted 
to  this  problem. 

The  applications  of  simulation  techniques  are  manifold.  They  range  from 
strategic  or  tactical  operations  to  management,  to  simply  system  operation. 

They  provide  a  means  by  which  the  analyst  can  handle  large  numbers  of  variables, 
mathematically  intractable  relationships,  and,  most  important,  uncertainties 
and  alternative  steps. 

3.2  QUEUING  THEORY 

Queuing  problems  may  develop  whenever  there  are  demands  for  service  from 
a  number  of  more  or  less  independent  sources.  Queuing  theory  is  a  technique, 
based  on  probability  theory,  that  supplies  a  means  for  mathematical  analysis  of 
this  class  of  problems. 
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Examples  of  queuing  (or  waiting  line)  situations  are  message  flow  in  a 
communications  center,  customer  servicing  at  a  repair  facility,  and  flow  of 
traffic  through  a  bottleneck.  Many  factors  must  be  considered  in  the  analysis 
of  queuing  problems.  Among  these  are*: 

.  The  probability  distribution  underlying  arrival  times 
.  The  probability  distribution  underlying  servicing  times 
.  The  number  of  waiting  lines 
.  The  number  of  servicing  facilities 
•  The  queue  discipline 


With  knowledge  of  these  factors,  the  analyst  can  often  predict  such 
important  results  as  the  average  length  of  the  waiting  line  and  the  average 
idle  time  for  a  service  facility  during  any  specified  time  interval. 


The  utility  of  this  method  can  be  demonstrated  by  an  example.  Messages 
arrive  at  a  communications  center  on  the  average  once  every  10  minutes  and 
with  Poisson  distribution: 


PT(n) 


(wr)n  e"XT 


The  service  times  for  processing  the  messages  are  assumed  to  be  exponentially 


distributed 


P(ts)  = 


with  means  of  3  minutes. 


The  questions  to  be  answered  are: 

(1)  What  is  the  average  number  of  messages  in  the  communications  center? 

(2)  Whet  is  the  average  length  of  the  queue  that  may  form? 

(3)  Assuming  that  another  message  clerk  will  be  put  on  when  a  message 
would  have  to  wait  at  least  3  minutes  before  being  processed, 

what  higher  rate  of  arrivals  can  be  tolerated  before  another  man  must 
be  assigned? 


For  this  particular  type  of  a  queue  the  following  relationships  can  be 
derived  by  means  of  queuing  theory: 


•  Average  number  of  messages  in  the  communications  center  = 


where 

A  =  average  arrival  rate  =0.1  per  min. 
u  =  average  service  rate  =  0.33  per  min. 


Reproduced  by  permission  from  NAVAL  OPERATIONS  ANALYSIS:  Copyright  1968  by 
the  U.  S.  Naval  Institute,  Annapolis,  Mi.,  p,238. 
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•  Average  length  of  "non-empty"  queues  =  -li- 

p.-A 


•  Average  waiting  time  =  ,  . 

h(h->0 


The  answers  to  the  three  questions  posed,  then,  are: 

o.  l 

(1)  Average  number  of  messages  in  the  system  =  '1  T  0.43  messages 

0  33 

(2)  Average  length  of  non-empty  queue  =  g  \  =  1.43  messages 


(3)  Tolerable  arrival  rate 


A' 


0.33(0.33-0 


7”  =  3-»A '  =  0.16  messages  per  minute 


10  messages  per  hour 


To  obtain  some  insight  into  the  underlying  theory,  consider  the  simplest 
case  ""  that  of  the  single- server  queue  with  Poisson  arrivals,  just  discussed. 

The  number  of  units  in  the  system  is  found  by  developing  recursion  relationships, 
which  are  governed  by  the  previously  cited  factors.* 

Let  n  =  the  total  number  in  the  system  (the  number  being  serviced  plus  the 
number  in  the  queue),  and  Pn  =  the  probability  of  there  being  n  units  in  the 
system.  Assume  that  the  queue  discipline  is  such  that  an  arrival  moves  immediately 
into  the  service  area  if  the  area  is  vacant. 


The  probability  of  an  arrival  in  a  small  time  increment.  At,  is  Mt. 

The  probability  of  a  serviced  unit  leaving  in  the  interval  t,  t  +At  is: 

0  if  no  units  are  in  the  system  at  t 
nAt  if  there  are  one  or  more  units  in  the  system  at  t 

The  probabilities  of  more  than  one  arrival  or  service,  or  both,  occurring 

2 

in  the  interval  are  taken  to  be  zero  since  they  are  proportional  to  At  or  higher. 

Consider  the  following  two  conditions: 

(1)  0  units  in  system  at  t  +  At 

(2)  0  units  at  time  t,  no  arrivals  in  At  +  1  unit  at  t  and  1  service 
completed  i  n  At 

These  two  events  are  equivalent  and  thus  their  probabilities  of  occurrence 
must  be  the  same. 

Thus  PQ  =  P0  (1  -  hat)  +  P1  tiAt,  or  ?1  =  £  PQ 


*M.  Sasieni,  et.  al..  Operations  Research  —  Methods  and  Problems.  John  Wiley 
and  Sons,  Inc.  1959?  p.  TZ5~. 
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To  obtain  ?2>  apply  the  same  procedure;  the  result  is. 

1-(A  +  p)  At 


P1  "  P1 


+  Po  (Mt)  +  P2  (pAt) 


o  r 


p  =  -  P,  -  -  P„  . 

II  1  LL  O  |J1 


—  H 
r2  IX 


This  can  then  be  generalized  to 


Pn  =  —  ^  P  -  -  -  P n  Pj  n  ^  2 
n  lx  n-1  p,  n-2' 


and,  by  induction,  this  can  be  written  as 


pn  *  &n  V  “  2  0 


However, 


so  that 


but 


Thus 


l  = 

n=o 


1 


Po  -  1 


p 


n 


I  Pn  '  1  '  I 
n=l 


=  (i  -  h  (A)n 


Once  the  Pn  are  determined,  relationships  such  as  those  used  in  the  previous 
example  can  be  found.  As  the  arrival  and  service  distributions  become  more 
complicated,  the  number  of  waiting  lines  and  service  facilities  increases,  or 
the  queue  discipline  becomes  more  complex,  and  the  associated  mathematics  becomes 
correspondingly  more  difficult. 

When  the  mathematics  becomes  too  complex  for  a  closed  analytical  solution 
or  too  costly,  the  approach  employed  is  Monte  Carlo.  To  Illustrate,  consider 
the  simple  case  of  two  sequential  service  facilities,  each  performing  to  some 
known  but  not  necessarily  simple  distribution.  Similarly,  the  arrivals  have 
some  known  but  not  necessarily  simple  distribution.  For  simplicity,  again  assume 
the  same  queue  discipline  as  previously.  The  desired  answer  is  the  expected 
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time  spent  in  the  system.  By  applying  Monte  Carlo  techniques  to  the  three 
distributions  for  each  item  in  the  system  and  keeping  track  of  where  the  item 
is  throughout  the  system,  the  time  each  item  spends  in  the  system  can  be  deter¬ 
mined  and.  by  simple  averaging,  the  resulting  expected  time  in  the  system  can 
be  ascertained  (subject,  or  course,  to  statistical  confidence  requirements). 

In  this  manner,  highly  complicated  systems  with  many  paths  and  serving 
points,  varied  distributions,  and  queuing  disciplines  can  be  analyzed. 

3.3  SEQUENCING  AND  MARKOV  PROCESSES 

Sequencing  is  related  to  the  order  in  which  units  requiring  servicing  are 
serviced.  It  applies  to  a  class  of  problems  in  which  the  measure  of  effectiveness 
is  a  function  of  the  sequence  or  order  in  which  a  number  of  tasks  are  performed. 
Sequencing  problems  fall  into  two  groups*:  (l)performing  n  tasks,  each  of  which 
requires  processing  on  some  or  all  of  m  different  machines;  and  (2)  processing 
a  list  of  n  tasks  with  m  machines,  with  the  decision  of  the  n+1  task  being  made 
at  the  completion  of  the  nth  task. 

Unfortunately,  both  types  of  problems  are  exceedingly  difficult;  at  present, 
solutions  are  known  only  for  some  special  cases  of  the  first  type. 

A  classical  example  of  the  sequencing  problem  is  the  "Traveling  Salesman" 
problem  in  which  the  salesman  must  visit  a  series  of  locations,  stopping  at  each 
location  only  once  and  returning  to  his  starting  point  at  the  conclusion  of  his 
travels.  An  analogous  operations  problem  is  the  selection  of  messenger  routes 
within  a  division. 

A  further  potential  application  is  to  use  sequencing  as  a  management  tool 
in  the  development  of  a  complex  system  requiring  numerous  tasks  with  various 
facilities  or  resources.  The  objectives  are  to  determine  the  optimum  use  of 
the  facilities  through  proper  sequencing  of  the  tasks  performed. 

To  illustrate  the  technique,  consider  a  messenger-route  problem  in  which 
five  stops  are  to  be  made  and  the  requirement  is  to  find  the  route  involving  the 
minimum  total  distance  to  travel**.  For  this  type  .of  problem  there  are  (n-  1)1 
subsets  that  must  be  searched  for  a  solution  --  in  this  case,  4|  or  24. 

This  problem  could  also  be  viewed  as  an  allocation  problem,  but  complicated 
with  the  added  constraint  that  the  messenger  must  not  pass  the  same  point  twice. 


Ibid,  p.  250. 
**Ibid}  p.  264. 
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Assume  the  distance  matrix  on  the 
left,  where  the  entries  (R's) 


represent  the  distances  from  point  to 
point.  (Note:  the  =  R^,  though 


in  general  cases  this  need  not  be  so. 
Also,  the  Rj^'s  have  been  assigned 
infinite  values  to  remove  them  from 


the  problem.  ) 


To  obtain  a  solution,  first  manipulate  the  matrix  as  follows: 


This  is  accomplished  by  the  following  steps: 

(1)  Identify  zeros  and  introduce  additional  zeros. 

(2)  Mark  unassigned  rows. 

(3)  Mark  columns  having  zeros  in  marked  rows. 

(4)  Mark  rows  having  assignments  in  marked  columns. 

(5)  Repeat  (3)  and  (4). 

(6)  Draw  line  through  unmarked  rows. 

(7)  Draw  line  through  marked  columns. 

(8)  Select  smallest  unmarked  element;  add  it  to  intersections;  subtract 
from  unmarked. 

The  smallest  value  is  1,  and  the  new  matrix  is  as  follows: 


00 

0 

3 

4 

X 

X 

oo 

X 

4 

□ 

3 

X 

CO 

0 

5 

4 

4 

0 

00 

2 

0 

X 

5 

2 

00 
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Identify  zeros  (the  allocation  problem  has  now  been  solved).  Check  the 
sequence:  A  -  B  -  E  -  A  (No  route  solution). 

Try  the  next  best  solution  by  expanding  about  the  smallest  non-zero  element, 
which  i  s  2. 


The  sequence  is  now  A-B-C“D“E“A.  at  a  cost  of  2  miles  above  the 
minimum  allocation.  This  is  a  valid  solution.  The  length  of  the  route  is  then 
2+3+4+5  +  1=15  miles.  (Note  if  the  expansion  had  been  performed  about 
the  other  2  element,  it  would  have  merely  reversed  the  order.)  This  solution 
is  a  minimum  route. 

The  behavior  of  a  system  that  has  discrete  states  with  probabilistic 
transitions  among  the  states  can  be  represented  by  what  is  known  as  a  Markov 
process  ""  a  conditional  process  in  which  the  next  transition  depends  on  the 
state  the  system  is  in  and.  for  some  types  of  process,  on  the  n  preceding  states 
as  well.  In  this  sense,  it  is  similar  to  sequencing.  However,  the  goal  here 
is  to  describe  the  system's  behavior  statistically  in  terms  of  its  transitions 
and  Its  ability  while  in  each  state. 

3.4  INVENTORY  AND  REPLACEMENT 

Inventory  can  be  defined  as  the  physical  stock  of  goods  kept  on  hand  by  an 
organization  to  promote  the  efficient  running  of  its  affaire.  The  costs  associated 
with  maintaining  an  inventory  are  normally  placed  in  three  categories: 

(1)  The  cost  of  ordering  goods 

(2)  The  costs  of  holding  goods  in  inventory 

(3)  The  costs  of  incurring  shortages 

The  problem  facing  the  decision-maker  is  twofold: 

(1)  Fhv  often  should  he  replenish? 

(2)  Hxv  much  should  he  replenish? 

Inventory-control  theory  is  a  mathematical  approach  to  finding  the  optimum 
re-order  time  and  quantity  and  is  usually  based  on  the  values  that  will  minimize 
the  overall  cost  of  maintaining  the  inventory. * 

"Reproduced  by  permission  from  NAVAL  OPERATIONS  ANALYSIS  Copyright  1968  by  the 
U  S.  Naval  Institute,  Annapolis,  Md.,  p.240. 
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The  system  can  thus  be  thought  of  as  consisting  of  outputs  (demands  for 
the  goods)  and  inputs  (replenishment  of  the  goods). 

Input  and  output  can  be  described  in  terms  of  their  respective  rates 
a(t)  and  b(t),  and  the  inventory  level  at  any  time  t  is  simply 


I(t)  =  I0 


a(t)  -  b(t) 


dt 


The  input  and  output  rates  do  not  necessarily  have  to  be  finite  but  their 
time  integrals  must  be  finite. 


The  Inputs  and  outputs  are  critical  factors  in  solving  inventory  problems. 

Most  inventory  models  include  one  of  the  three  types  of  assumptions  regarding 
these: 

(1)  Continuous  in  time 

(2)  Discrete  and  equidistant  in  time 

(3)  Discrete  and  irregular  in  time 

The  inventory  problem  is  also  classified  in  terms  of  the  amount  of  knowledge 
regarding  these  factors;  i.e.,  the  factors  are  either  known  and  hence  deterministic, 
or  unknown  and  hence  probabilistic. 

Further,  if  the  parameters  are  constant  with  time,  the  problem  is  said  to 
be  static;  if  they  are  not,  the  problem  is  dynamic,  with  corresponding  increases 
i  n  mathematical  difficulty. 

The  inventory  system  itself  may  be  complex,  with  various  stations  (series 
and  parallel)  and  various  levels.  In  addition,  the  links  between  stations 
can  vary  in  form  (single,  alternative,  fusion).  Taken  as  a  whole,  the  system  can 
be  considered  analogous  to  a  network. 

A  final  consideration  to  be  discussed  concerning  inventory  model8  is  the 
delay  factor  between  re-order  and  replenishment.  The  principal  difference  that 
results  from  including  the  delay  is  that  further  depletion  of  inventory  between 
the  decision  and  the  arrival  of  replacements  must  be  considered. 


To  illustrate  the  technique,  consider  the  simple  example*  of  a  single  station 
that  has  a  uniform  demand  rate  b(t)  =  B  units  per  unit  time.  Units  are  re-ordered 
every  T  days  and  the  re-order  cost  is  C.  There  is  no  appreciable  delay  in 
filling  orders,  so  that  the  practice  is  to  re-order  whenever  the  inventory  is 
zero. 


*Sasieni,  ojd.  clt ,  ,  p.  71. 


3-10 


AMCP  706-191 


The  cost  of  inventory  is  assumed  to  be  proportional  to  the  number  of  units 
held  and  the  length  of  time  they  are  held.  Thus  the  Inventory  cost  here  is  C^IT, 
where  is  the  unit  holding  cost  per  unit  time  and  I  is  the  amount  in  the  inven¬ 
tory.  The  amount  reordered  every  T  must  be  BT,  and  the  Inventory  holding  costs 

are  thus 

m  rp 

CI  =  C1  f  I  dt  =  CjB  J  (r'b-bt)dt  =  £  c^T2 
^  o  Jo 


The  average  total  cost  per  unit  time  is 

Ct  =  £  ^BT  + 


For  a  minimum  cost,  differentiate  with  respect  to  T  and  equate  to  zero: 


dOj 
cTF"  = 


0 


o  r 


T  = 


2C„ 

_ £  days 

\/ClB 


The  quantity  to  be  re-ordered  Is  then 

Q  =  HT  =  /  2BCr 

J  — 


and  the  minimum  average  total  cost  is 


*  J2C1  B  °r 

Replacement  theory  is  concerned  with  situations  in  which  system  performance 
deteriorates  with  time  and  the  system  can  be  restored  to  its  initial  condition 
through  some  kind  of  action.  The  problem  is  to  determine  when  these  actions 
should  be  taken. 
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3.5  LINEAR  AND  DYNAMIC  PROGRAMMING 

3.5  .1  Linear  Programming 

Linear  programming  is  used  to  determine  the  values  of  a  set  of  variables  in 
a  linear  equation  that  produce  an  extrema  in  the  objective  while  the  variables 
are  subject  to  a  set  of  linear  constraints. 

Linear  programming  problems  generally  fall  into  two  categories,  assignment 
and  transportation,  although  the  latter  is  actually  a  generalization  of  the  former. 
Assignment  problems  generally  deal  with  distribution  between  a  number  of  alter¬ 
natives  in  such  a  manner  as  to  maximize  or  minimize  the  total  worth  or  objective. 
Transportation  problems  generally  deal  with  routing  of  units  between  a  number  of 
sources  and  receivers  in  such  a  manner  as  to  maximize  or  minimize  the  worth  of  the 
operation. 

However,  the  problems  need  not  concern  only  assignment  or  transportation  for 
linear-programming  techniques  to  be  applied.  Any  problem  that  can  be  formed  as 
optimizing  a  linear  expression  subject  to  linear  constraints  can  be  treated. 

A  mathematical  representation  of  the  linear-programming  problem  is  simply 

N 

“max  "  X  a»X“ 

n=l 


subject  to 


xn  g  0  and 


N 


S  d 


m=0,  1,  .  ,,M 


There  are  a  number  of  variations  in  forming  these  relationships,  such  as  the 
direction  of  the  inequality  and  whether  the  purpose  Is  maximizing  or  minimizing. 


A  number  of  techniques  have  been  developed  for  solving  linear-programming 
problems.  Two  of  these,  one  graphical  and  one  analytical,  are  treated  below. 


Consider  the  problem  of  two  types  of  helicopters,  A  and  B.  and  the  following 
circwnstarices: 

•  Type  A  carries  30  troops;  B  carries  20  troops. 

.  There  are  fifty  pilots  available. 

.  Type  A  requires  two  pilots;  B  requires  one  pilot. 

.  There  are  40  of  the  A-type  helicopter  and  20  of  the  B  type. 

.  The  objective  Is  to  move  the  maximum  number  of  troops. 
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The  above  statements  can  be  changed  into  the  following  mathematical 
expression: 

Maximize  30XA  +  20Xg  are  the  number  of  the  types  A  and  B  used, 
subject  to  the  following  constraints: 

(1)  2Xa  +  Xb  650 

(2)  XA  s  40 

(3)  XB  £  20 

First  consider  the  graphical  solution  shown  below. 


Objective 

Function 


The  procedure  followed  to  find  the  optimal  solution  is  as  follows: 

•  Plot  inequalities  -  Equations  1,  2,  and  3. 

.  Note  region  allowed  by  each  -  inside  crosshatched  lines. 

.  Note  solution  region. 

•  Plot  objective  function. 

•  Move  objective  function  (parallel  to  itself)  away  from  the  origin. 

•  Note  maximum  distance  point  (last  point  in  the  solution  region  that  the 
objective  function  touches). 

The  solution  here  is  to  use  20  of  B.  15  of  A. 

It  should  be  noted  that  constraint  number  2  could  have  been  neglected  with¬ 
out  changing  the  solution  in  this  example. 

The  graphical  method  is  a  quick  and  easy  method  for  solving  linear-program¬ 
ming  problems,  provided  there  are  only  two  variables.  For  three  or  more  varisbles, 
analytical  techniques  are  required  because  the  solution  space  is  no  longer  two- 
dimensional. 
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The  analytic  technique  described  here  is  the  simplex  technique.  theory 

behind  it  i s  complicated,  but  the  application  is  relatively  simple,  although 
tedious.  (The  technique  is  readily  programmed  for  solution  by  computers.) 


To  solve  the  sample  problem,  it  is  first  necessary  to  write  the  system  of 
inequalities  (constraint  equations)  as  equalities,  by  introducing  a  set  of  slack 
variables  --  and  ; 

2Xa  +  XB  +  Sx  =  50 
XA  +  S2  =  40 
XB  +  S3  =  20 

Then,  rewrite  the  objective  function  as 

-30Xa  -  20Xb  +k  =  0  (4) 


(1) 

(2) 

(3) 


where  M  represents  the  term  to  be  maximized. 

Now  construct  a  matrix  of  the  coefficients  of  Equations  1  through  4: 


XA.  Xr 

2  1 

1  0 

0  1 

-30  -20 


1  0 
0  1 
0  0 
0  0 


S3  M  N 

0  0  50 

0  0  40 

1  0  20 

0  10 

t 

Objective 

Column 


Objective  Row 


Designate  the  M  column  as  the  objective  column  and  the  last  row  (objective 
function)  as  the  objective  row.  A  feasible  solution  is  present  when  at  least 
two  of  the  columns;  other  than  the  M  and  N  columns,  contain  exactly  one  1  and 
all  the  other  entries  are  zeroes  and  all  the  l's  are  not  in  the  same  rows. 


For  the  matrix  shown,  there  is  a  feasible  solution:  s  =  50,  Sg  =  40, 

=  20,  thus  making  xA,  xB,  M  —  0.  However,  this  is  not  the  optimum  solution; 

i.e.,  no  troops  are  moved. 

To  check  whether  the  solution  is  optimum,  examine  the  objective  row  to  see 
if  there  are  any  negative  entries.  if  there  are  no  negative  entries,  the  solution 

is  the  optimum  one.  In  this  case,  there  is  a  -30  and  a  -20;  thus  the  solution  is 

not  optimum,  and  the  following  procedure  is  carried  out: 

(I)  Determine  the  most  negative  element  in  the  objective  row  and  identify 
its  column  (the  XA  column  in  this  problem)  . 
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(2)  Divide  each  positive  element  in  the  selected  column  into  its 
corresponding  row  value  in  the  N  column. 

^3 j  Circle  the  element  producing  the  smallest  ratio  (the  element  2 
in  the  column,  which  has  a  ratio  of  25).  This  is  known  as 
the  "pivot"  number. 

(4)  Next  normalize  the  pivot  number  and  make  all  other  entries  in 
the  pivot  column  zero.  This  is  done  by  first  dividing  every 
element  in  the  pivot  row  by  the  pivot  number  to  obtain  a  new, 
normalized  pivot  row.  Second,  for  each  other  row,  multiply  the 
normalized  pivot  row  by  the  negative  of  the  corresponding  pivot- 
column  element,  and  add  the  two  rows  to  obtain  a  new  row  having 
a  zero  in  the  corresponding  position  in  the  pivot  column. 

For  this  problem,  the  normalizing  is  accomplished  by  dividing  the  pivot  row 
by  two.  then  multiplying  the  new  pivot  row  by  -1  and  adding  element  by  element, 
to  row  2  to  obtain  a  new  row  2.  Row  3  is  already  0  in  the  pivot  column,  there¬ 
fore,  nothing  has  to  be  done  to  it.  Finally,  multiply  the  normalized  row  by  30 
and  add  it  to  row  4.  The  resulting  matrix  is  as  follows: 


XA  XB 

S1 

S2 

M 

N 

1  1 

i 

0 

0 

0 

25 

0  -1 

-i 

1 

0 

0 

15 

0  1 

0 

0 

1 

0 

20 

o  -5 

15 

0 

0 

1 

750 

This  procedure  is  repeated 

until  there  arc  no 

longer  negative  entries  in 

the  objective  column,  and  the  resulting 

solution  is 

;  optimal. 

The  next  pivot  element  is  i 

row  3 ,  Xg 

column.  The  resultant  matrix  is  as 

follows  : 

XA  XB 

S2 

a* 

M 

N 

1  0 

1 

2 

0 

i 

“  2 

0 

15 

0  0 

1 

“2 

1 

1 

”2 

0 

25 

0  1 

0 

0 

1 

0 

20 

0  o 

15 

0 

5 

1 

850 

Since  there  is  no  longer  a 

negative 

element  in  the 

objective  row,  the 

solution  is  optimal  and  equal  to  XA  =  15 

’  X B 

=  20, 

and  M  =  850. 

A  necessary  condition  for 

the  formulation  of  iinear  programming  problems 

is  a  linear  set  of  objective  functions  and  constraints. 

However,  there  are  many 

situations  in  systems  analysis 

—  when  one  or  more 

of  the  functions  are  expressed 

as  a  product  equation  in  the  variables  - 

-  in 

which 

this 

technique  can  be  applied 
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but  not  all  the  equations  are  linear.  This  often  occurs  when  kill  probabilities 
of  targets  are  being  determined.  In  such  a  case,  the  equation  is  linearized  by 
converting  to  the  logarithim  of  the  function  and  optimizing  on  the  log  (which  is 
monotonic  to  its  antilog)  . 


To  illustrate,  consider  the  case  in  which  there  are  two  types  of  weapons  and 

three  types  of  targets,  with  P,  .  being  the  kill  probability  of  the  target 

th 

type  by  the  i  weapon  type.  The  objective  is  to  determine  the  allocation  of 
weapons  to  targets  to  maximize  kill  probability  for  at  least  one  target.  This 
is  the  same  as  minimizing  the  probability  of  not  killing  any  target.  Let  p  denote 
this  probability.  Thus 

p  =  (1-PU)NU  (1-P12)N12  (i-p13)N13  (i-p21)N21  (i-p22 


)N22  (l-p23) 


N23 


Taking  the  logarithim 

results  in 

i=2 

i=2 

3=3 

j=3 

Log 

P  =  £  Ni(J  Log  (1-P^) 

-  ■  I  Nu 

1  =  1 

1  =  1 

3=1 

3=1 

This  can  be  minimized  by  maximizing  Log  =  -Log  P.  Thus  the  objective 
function  is  to  maximize  .  . 

i=2 

3=3 

L°g  P  =  ^  Nj.j  *ij 

i=l 

3=1 

subject  to  the  constraints  cited. 

3.5.2  Dynamic  Programming 

In  dynamic  programming,  there  are  no  restrictions  on  the  set  of  equations, 
nor  are  there  any  general  algorithims  for  problem  solution.  Dynamic  programming 
was  developed  as  a  means  of  studying  decision  processes  and  determining  the 
sequence  of  decisions  that  results  in  optimizing  a  predetermined  objective  func¬ 
tion  . 

In  defining  this  sequence.  Bellman*  (who  is  the  originator  of  this  method) 
set  forth  a  principle  of  optimality  stating  that  an  optimal  policy  was  one  which 
insured  that  each  decision,  in  the  sequence  of  decisions  was  the  optimum  decision 
with  respect  to  the  conditions  resulting  from  the  prior  decisions. 

Some  recent  applications  of  dynamic-programming  techniques  include: 

.  Determining  thrust-control  policies  and  fuel  consumption  regimes  for 
putting  satellites  into  specified  orbit  altitudes  with  maximum  horizontal 
components  of  velocity 

* Bellman,  R,,  Dynamic  Programming,  Princeton  University  Press,  Princeton,  N.J. , 
1957. 
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.  Determining  optimum  staging  ratio  for  missiles  (how  many  booster  stages  of 
what  sizes  result  in  most  efficient  missiles) 

•  Establishing  optimum  inventory  control  schemes  for  interacting  inventories 
at  different  locations 

In  summary,  dynamic  programming  selects  the  optimum  sequence  of  decisions  to 
establish  a  policy  that  will  bring  a  maximum  return. 

3.6  GAME  THEORY 

Game  theory  is  a  mathematical  theory  of  decision-making  by  contestants  with 
various  strategies.  Originally,  it  was  developed  to  handle  business  and  economic 
problems;  however,  it  has  found  extensive  application  in  military  systems  and 
operations  analysis. 

The  theory  is  defined  as  a  mathematical  demonstration  that  if  opposing 
interests  act  rationally  to  achieve  desired  ends  that  can  be  set  forth  validly 
in  a  numerical  scale  of  expected  returns,  returns  that  vary  according  to  the 
success  of  various  plans,  the  appropriate  strategy  for  each  side  can  be  deduced 
mathematically.* 

The  following  terms  are  used  in  discussing  game  theory: 

•  Game  —  the  set  of  rules  that  define  what  can  or  cannot  be  done,  the 
size  of  the  bets  or  penalties,  and  the  payoff  methods 

•  Play  of  the  Game  —  one  complete  ran  through  the  game,  including  payoffs 

•  Zero-Sum  Game  --  a  game  in  which  the  gains  of  one  side  equal  the  losses 
on  the  other 

.  Strategy  --  a  plan  of  action  that  is  complete  and  ready  to  use  before  the 
game  commences 

.  Person  -  one  of  the  opposing  interests 

•  N-Sided  Game  —  N  opposing  persons 

.  Pure  Strategy  —  a  decision  always  to  follow  a  particular  course  of 
action 

•  Mixed  Strategy  --  a  decision  to  choose  a  course  of  action  for  each  play 
in  accordance  with  some  probability  distribution 

•  Value  —  the  expected  gain  in  one  play  of  the  game  with  all  players 
using  stable  optimum  strategies 

A  competitive  game  has  several  characteristics  worth  noting**: 

•  There  is  a  finite  number  of  persons. 

•  Each  person  has  a  finite  set  of  strategies. 

*  Reproduced  by  permission  from  NAVAL  OPERATIONS  ANALYSIS:  Copyright  1968  by  the 
U.  S.  Naval  Institute,  Annapolis.  Md..  p.30. 

**Sasieni,  op.cit.,  p.156. 
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.  Strategy  choices  are  made  simultaneously. 

.  There  i  s  an  outcome  of  a  play  that  determines  a  set  of  payoffs  to  each 
player  . 


The  simplest  game  is  the  two-person/zero-sum  game.  This  game  is  iTlustratea 
by  a  problem  in  which  the  commander  of  a  unit  is  planning  to  employ  a  communi¬ 
cation  system  and  he  has  four  candidate  systems,  while  the  enemy  commander  has 
five  types  of  jamming  equipment  to  employ.  The  payoff  for  each  combination  of 
communication  system/jammer  is  measured  in  terms  of  the  expected  error  probability. 
The  problem  is  to  select  the  strategy  to  be  employed  by  each  commander.  Assume 
that  through  analysis  the  following  payoff  matrix  was  determined: 


Jamming 
n.  System 

C  ommu  n  i  c  a  t  i  on'-\^ 
System  N, 

I 

II 

III 

IV 

V 

Maximum 

Error 

A 

0.1 

0.7 

0.8 

0.6 

0.4 

0.8(111) 

B 

0.4 

0.5 

0.6 

0.3 

0.3 

Q(III) 

C 

0.8 

0.6 

0.7 

0.7 

0.1 

0.8(1) 

D 

0.4 

0.4 

0.8 

0.8 

0.2 

0.8(111) 

Minimum  Error 

O.l(A) 

0.4(D) 

f^KB) 

0.3(B) 

0.1(C) 

The  commander's  objective  is  to  select  the  strategy  that  gives  him  the 
minimum  error  probability,  while  the  opposition  desires  to  choose  the  strategy 
that  maximizes  the  error  probability. 

The  approach  taken  is  to  examine  each  communications  strategy  and  determine 
which  results  in  the  poorest  return,  thus  reflecting  the  poorest  expected  return. 
This  information  is  shown  to  the  right  of  the  matrix.  Similarly,  each  jamming 
strategy  is  examined  for  its  worst  case,  and  the  values  are  shown  below'  the  matrix. 

Each  commander  then  selects  the  best  of  his  worst  solutions  as  a  strategy 
(circles  appropriate  values).  Thus,  in  the  rows,  look  for  a  Min-Max  solution  and 
in  the  columns,  a  Max-Min  solution. 

From  the  matrix  observe  that  communications  system  B  and  Jammer  system  III 
would  be  chosen. 

Note  that  in  this  case  both  strategies  are  defined  by  the  same  element. 

Such  a  solution  is  known  as  a  Saddle  Point,  and  the  resulting  strategies  as  Pure 
Strategies.  The  value  of  the  game  is  0.6,  and  if  either  side  uses  a  different 
strategy,  his  expected  return  will  be  reduced  (in  this  case  the  error  probability 
would  increase  to  the  communicator  or  decrease  to  the  jammer). 

If  no  Saddle  Point  occurs,  the  best  strategies  are  mixed  strategies,  and 
the  game  solution  is  the  set  that  maximizes  the  expected  return.  To  illustrate 
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this,  consider  the  same  problem  as  above,  but  with  the  (B,  111)  element  changed 
to  say,  0.1.  The  new  matrix  is  as  follows: 


n.  Jamming 

System 

Oommunlc at iorK. 
System 

I 

II 

III 

IV 

V 

Maximum 

Error 

A 

0.1 

0.7 

0.8 

0.6 

0.4 

0.8(111) 

B 

0.4 

LO 

o 

O.l 

0.3 

0.3 

@11) 

C 

0.8 

0.6 

0.7 

0.7 

0.1 

0.8(1) 

D 

0.4 

0.4 

0.8 

O.Q 

0.2 

0.8(111,  IV) 

Minimum  Error 

o.i(a; 

@d) 

0.1(B) 

0.3(B) 

0.1(c) 

From  the  matrix  observe  that  system  B  represents  the  optimum  strategy  and 
jammer  II  represents  the  best  counter-strategy.  However,  there  is  no  Saddle 
Point;  hence,  pure  strategies  no  longer  exist  • 

The  first  step  to  the  solution  of  this  problem  is  to  try  to  reduce  the 
dimensionality  of  the  game.  It  can  be  seen  that  no  strategy  dominates  within 
the  rows.  However,  within  the  columns,  column  IV  dominates  column  V  in  every 
row.  Hence,  column  V  is  dropped  from  further  consideration.  Of  the  remaining 
matrix,  row  B  is  now  seen  to  dominate  row  C.  Carrying  the  elimination  procedure 
to  its  limit  results  in  the  following: 


II  IV 


0.5 

0.3 

0.4 

0.8 

Nw  let  X  =  (Xg,  Xjj)  and  Y  =  (Yjj,  Y Iv)  equal  the  optimum  mixed  strategies 
of  the  two  sides. 

The  gain  is  now  a  random  variable,  g,  and  the  expected  value  to  each  side 
i  s 


E(g:  x- y)  - 1  *u  xi 
U 

E(-g;  x- y)  ’  Z(-aij>Vj 
ij 

Let  V1  and  V2  represent  the  expectation  of  each  side,  which  is  to  be 
optimum. 
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Thus  strategies  must  be  chosen  so  that: 

X  ,  Y)  s  V1  given  XQ  is  optimum  communication  system  strategy 

E(-g;  X,  Yq)  s  Vg  given  Y0  is  optimum  jammer  system  strategy 

Since  the  game  is  zero-sum,  V1  =  -V2  and  E(g;  XQ,  Y0)  =  V,  which  means  that 
if  both  sides  use  their  optimal  Min-Ma\  strategies,  their  achieved  gains  coincide 
with  their  Min-Max  expected  gains  (or  in  this  case  their  maximum)  is  being  mini¬ 
mized 

Thus 

E(gj  XQ,  Y)  s  V 
E(g;  X,  Yo)  *  V 

Substituting  for  the  expected  payoffs  yields 

Y^O.5^  +  0.4Xd)  +  Y^O.SXg  +  0.8Xd)  s  V 

Xb(0.5Yii  +  0.3YIV)  +  Xd(0.4Yi;i;  +  0.8YIV)  *  V 

In  addition, 

XB  +  XD  “  1 

YII  +  YIV  =  lf  and 

X±  and  Yj  s  0 

The  above  inequalities  imply  the  following  relationships: 

O.SXg  +  0.4Xd  s  V 
0.3Xg  +  0.8XD  £  V 

°-5Yii  +  0,3YIV  s  v 
0.4YIX  +  0.8YIV  6  V 

Thus  there  are  five  unknowns  and  ten  relationships.  Not  all  elements  can 
be  zero,  but  it  is  possible  for  equalities  to  hold  in  the  four  Inequalities. 

Thus 

O.SXg  +  0.4XD  =  V 
0.3Xb  +  0.8xd  =  V 
O.SYjj  +  0.3YJV  =  V 
o.4yI3;  +  o.8yiv  =  V 
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Xb  +  Xjj  -  1 


Solving  this  system  leads  to 


LII 


+  Y. 


IV 


1 


Xb  =  2/3 
XD  =  1/3 
YII  -  5/6 
YIV  ■  1/6 


Thus  by  using  communications  set  B  two-thirds  of  the  time  and  set  D  the  remainder, 
the  commander  w  i  1 1  realize  a  payoff  of  0.467,  whereas  if  he  were  to  have  stayed 
with  his  best  pure  strategy,  he  would  have  been  sure  of  no  better  than  0.5. 
Similarly,  by  jamming  with  jammer  number  II  five- sixths  of  the  time  and  jammer  IV 
the  remainder,  the  enemy  is  assured  of  a  payoff  of  0.467,  whereas  by  following  his 
pure  strategy,  he  would  not  have  been  sure  of  doing  better  than  0.4. 


Games  with  large  matrices  are  often  tedious  to  solve  by  use  of  the  technique 
just  described.  However,  it  is  relatively  simple  to  obtain  an  approximation  of 
the  exact  solution* 


Consider  the  reduced  matrix  of  the  sample  problem: 


2.8 


-2/3 
-  1/3 


1  1 

3/6  1/6 


*Sasienij  o £.  clt . .  p.  170. 
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Solution  rules  are  as  follows: 

(1)  Select  the  row  and  place  under  matrix  (0.5,  0.3). 

(2)  Circle  smallest  value  in  the  row  and  write  corresponding  column 
to  right  of  the  matrix  (0.3,  0.8). 

(3)  Circle  largest  value  in  column  and  write  corresponding  row  and  add 
to  last  row* 

(4)  Circle  smallest  value  in  row  and  write  corresponding  column  and  add 
to  last  column* 

(5)  Repeat  process  N  times. 

The  approximate  strategies  after  N  iterations  are  then  the  number  of  circled 
values  divided  by  N  for  each  choice. 

Solving  this  system  leads  to  the  following  values: 


=  0.667 

*b 

=  0.667 

=  0.333 

XD 

=  0.333 

YII 

=  0.723 

YII 

=  0.833 

YIV 

=  0.277 

YIV 

=  0.167 

Note  that  the  values  are  of  the  right  order,  but  their  convergence  is  not 
particularly  rapid. 

The  upper  and  lower  bounds  for  the  game  can  be  determined  by  dividing  the 
highest  number  in  the  last  column  (8.4  after  18)  by  the  number  of  iterations,  and 
by  dividing  the  lowest  number  in  the  last  row  (8.3  after  18)  by  the  number  of 
iterations.  Thus,  for  this  example,  the  value  is 

0.45  s  V  s  0.468 

while  the  predetermined  answer  was-V  =  0,467. 

As  the  number  of  sides,  the  number  of  moves  per  play,  and  the  dimensionality 
increase,  the  complexity  of  the  game  solution  increases  correspondingly. 

3.7  INFORMATION  THEORY 

Information  theory  is  a  relatively  new  tool  to  the  systems  analyst.  Its 
initial  and  most  frequent  applications  have,  of  course,  been  in  communication 
system  problems.  However,  it  has  received  other  application  in  such  diverse  areas 
as  missile-guidance  and  maintenance-reporting-system  analyses. 


*  Settle  ties  by  choosing  the  opposite  value  from  preceding  time. 


3-22 


AMCP  706-191 


In  general,  the  theory  can  be  applied  to  any  situation  in  which  there  is 
uncertainty  to  be  reduced  or,  more  particularly,  where  there  is  a  source  that 
has  uncertainty  connected  to  an  information  sink  by  a  channel  that  may  perturb 
the  source  outputs. 

Thus  "information",  as  used  here,  does  not  represent  a  body  of  data,  but 
merely  the  amount  of  uncertainty  that  has  been  reduced. 

A  possible  application  of  information  theory  is  to  measure  effectiveness 
for  a  communication  system,  i.e,,  how  much  information  is  conveyed  by  the  system, 
or  how  fast  it  is  conveyed. 

For  a  quantitative  treatment  of  this  Subject,  several  concepts  must  be 
defined.  First,  information  is  defined  as  being  expressed  in  M-airy  units,  given 
by  I  =  -Log^  P^>  where  is  the  probability  of  having  selected  message  i  from  a 
source  containing  m  symbols.  The  most  frequently  used  case  is  the  binary,  in 
which  there  are  two  symbols  (1,  0).  Thus  for  a  message  consisting  of  a  single 
symbol,  the  information  is  expressed  in  binary  units  and  written  as 

I  =  Log2 

For  example,  the  information  conveyed  by  flipping  a  legitimate  coin  is 

I  =  -Logg(|-)  =  1  bit, 

while  the  information  conveyed  by  a  two-headed  coin  is 

I  =  -Logg(l)  =  0  bits; 
i.e.,  the  outcome  is  known  in  advance. 

An  important  concept  in  the  study  of  information  is  that  of  entropy;  simply 
stated,  this  is  the  average  uncertainty  of  a  source  or  message.  Thus 

=-pl  t°*m  P1 

i  i 

To  illustrate,  consider  the  entropy  continued  in  a  two-digit  message  (binary) 
having  the  following  message-population  distribution: 

Message  Probability  of  Sending 


11 

1/2 

01 

1/8 

10 

1/8 

00 

1/4 
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Thus 

H  =  i  Log2  2  +  1/8  Log28  +  1/8  Log28  +  k  Log2  4  =  1  3/4 
bits  of  information  on  the  average  is  transmitted  by  a  message  from  this  source. 

It  can  be  proved  that  the  maximum  entropy  is  achieved  when  a  1 1  of  the 
messages  are  equally  likely.  Further,  when  the  symbols  are  independent  with 
the  same  distributions,  the  entropy  of  a  message  of  length  n  is  n  times  the 
entropy  of  a  single  symbol. 

The  next  concept  to  be  defined  is  that  of  a  channel,  A  channel  is  described 
by  an  input  alphabet  A  and  an  output  alphabet  B  and  a  set  of  conditional  prob¬ 
abilities  (P(Bj|a1)  ]  ,  termed  the  channel  matrix,  that  are  the  probabilities 
of  receiving  message  Bj  given  A^  was  sent. 

To  illustrate,  consider  a  simple  binary  source  having  a  symmetric  channel  — 
that  is,  the  probability  of  an  error's  being  introduced  on  a  one  is  the  same  as 
the  probability  of  its  being  introduced  on  a  zero.  Symbolically  this  is  expressed 
as  follows: 


A  B 


where  the  channel  matrix  is: 

P  P 
P  P 


Associated  with  the  notion  of  a  channel  are  several  other  quantities  worth 
mentioning  : 

•  The  a  priori  entropy  of  A  =  H(A)  =  -  ^  P(a)  Log  P(a), 

A  ^ 

•  The  posteriori  entropy  of  A  =  H(A|bj )  =  -  ^  P(a  |bj  )  Log  P(A| ) . 

A 

•  Conditional  entropy  H(A|B)  =  ^H(A|bj)  P(bj),  which  is  called  the 

equivacation  ,  B 


•  Mutual  Information  (the  information  provided  by  the  observation  of  an 
output  symbol)  =  I  (A;B)  =  H(A)  -  H(A|b) 


H(A,B) 
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Some  interesting  properties  of  mutual  information  are  the  following: 

•  It  is  always  non-negative,  i,e,,  I(AjB)  2  0. 

•  If  channels  are  cascaded,  they  will  tend  to  leak  information,  i.e,, 

I(A;B)  S  I(AjC). 

•  Mutual  information  is  additive  i.e,,  l(A;B,C)  =  l(A;C)  +  I(AjB(c), 

Channel  capacity  is  a  measure  of  the  ability  of  the  channel  to  transmit 
information;  it  is  defined  mathematically  as  C  =  Max  I(AjB), 

PtaJ 

Channel  capacity  is  commonly  expressed  in  information  units  per  unit  of 
time.  The  notion  of  channel  capacity  leads  to  one  of  the  fundamental  theorems 
of  information  theory:  “If  the  average  amount  of  information  per  message  from 
a  source  is  H  and  the  channel  has  a  capacity  of  C,  then  it  is  possible  to  encode 
the  messages  so  that  they  may  be  transmitted  over  the  channel  at  a  rate  R  which 
has  a  maximum  value  of  C/H,"* 

If  the  concept  of  a  noisy  channel  is  now  introduced,  the  preceding  can  be 
modified  to  "if  the  rate  of  transmission  is  less  than  the  channel  capacity,  it 
is  possible  to  encode  a  message  for  transmission  so  that  an  arbitrarily  small 
percentage  of  errors  may  be  obtained.  "** 

The  preceding  discussion  concerns  discrete  messages .  However,  a  completely 
analogous  development  exists  for  the  continuous  case,  wherein  summations  are 
replaced  by  integrals  and  discrete  probabilities  by  density  functions,  Thus  the 
entropy  of  a  continuous  source  would  be  given  by*** 

r°° 

H  =  -j  p(x)  log  p(x)  dx 

•J—, 

Here  entropy  will  not  be  unique  but  will  depend  on  the  co-ordinate  system 
used  to  represent  the  variable.  However,  in  the  noisy-channel  situation,  it  is 
the  mutual  information  that  is  of  interest: 


I  (x;y)  =  H(y)  -  H(y  x) 


P  (y)  dy  - 


-J  P(x)  dx  J 


Px(y)log  Px(y )dy 


This  equation  represents  the  difference  in  entropies  --  one  term  representing 
the  received  signal  and  the  other  term  representing  the  effects  of  the  noise. 

Then,  as  long  as  both  terms  possess  the  same  units,  the  solution  will  be  unique 
and  hence  not  dependent  on  the  co-ordinate  system  employed. 


*C  ,  D .  FTagTe  et.  al,, 
Press,  I960,  p,  590. 

**Ibid . ,  p,  597 , 
***rbid. ,  p.606. 
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According  to  Shannon's  Theorem,  an  amount  of  information  per  sample  point 
can  be  sent  over  a  noisy  channel  as  given  by  the  maximum  of  the  equation  above. 
However,  the  channel  must  be  evaluated  in  terms  of  the  specific  channel  used. 

The  channel  is  restricted  by  the  bandwidth  available  and  the  power  available  for 
the  signal  waveform.  For  a  signal  of  average  power  P  in  the  presence  of  narrow- 
band  Gaussian  noise  of  average  power  N  and  bandwidth  W,  the  channel  capacity  is 
given  by  : 

C  =  W  Log  ( 1  + 

Thus  the  trade-off  between  power  and  bandwidth  is  shown, 

When  the  basic  relationships  pf  information  theory  discussed  above  are 
applied  to  a  systems-analysis  problem,  the  components  of  the  system  are  repre¬ 
sented  as  channels  with  appropriate  characteristics  and  the  inputs  and  outputs 
correspond  to  the  information  passed  by  the  system. 

To  demonstrate  the  application  of  these  techniques,  consider  a  slightly 
different  example  --  a  maintenance  system.  A  simple  system  experiences  three 
types  of  failures  and  exhibits  four  types  of  symptoms.  Analysis  of  symptom/failure 
frequency  data  yields  the  following  matrix,  where  each  element  is  the  number  of 
times  the  corresponding  failure/symptom  combination  was  experienced: 


^’"""--■^^Symptom 

Fa  i  lur 

S 

1 

s 

2 

s 

3 

S 

4 

Totals 

F1 

5 

4 

1 

0 

10 

F2 

2 

1 

2 

5 

10 

F3 

1 

2 

5 

2 

10 

Totals 

8 

7 

8 

7 

30 

The  first  step  in  the  analysis  is  to  convert  the  elemental  values  to  prob¬ 
abilities  . 


^'^'^^Syrr.p  tom 

Fa  1  lur 

S1 

S2 

S3 

S4 

Totals 

F1 

1/6 

2/15 

1/30 

0 

1/3 

F2 

1/15 

1/30 

1/15 

1/6 

1/3 

f3 

1/30 

1/15 

1/6 

1/15 

1/3 

Totals 

4/15 

7/30 

4/15 

7/30 

1 
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The  average  information  contained  in  a  symptom  is 

4 

H(S)  =  P(S±)  Log£  P( S± ) 

1=1 


_  -8  4 

=  -  —  Log£  - 


15 


15 


2_ 

15 


4og2 


2- 

30 


=  ^  (3.907-2.000)  +  ^  (4.907-2.807) 


-1^(1.907)+^  (2.100) 

=  1,997  bits  per  symptom 

Similarly,  the  entropy  contained  in  the  failures  is 

H(F)  =  P(Fj )  Log2  P(Fj)  =  -Log2  ^  =  1.585  bits  per  failure 
J*1 

The  joint  entropy  in  a  symptom  is  found  directly  from  the  symptom/failure 
matrix  (note  that  in  this  case  the  matrix  is  not  the  channel  matrix)  : 

4  *3 

H(S,F)  =  -£  £  P(Si'Pj)  P(S1,PJ) 

1-1  J-l 


"  Log2  F  "  h  L°g2  15  ~  TO  35  "  fe  Log2  15 

“  4-  (2.585)  +|j-  (2.907)  (4.907)  +  ^  Log  3.907 


=  3.213  bits 

The  information  transmitted  from  symptom  to  failure  (i.e,,  the  mutual  infor¬ 
mation)  is  given  by 

I(S,F)  =  H(S)  +  H(F)  -  H(S,F) 

(This  can  be  derived  from  the  earlier  expression  for  I  that  included  equivocation.  ) 

Thus, 

I(S,F)  =  1.997  +  1.585  -  3.213  =  0,369  bits 

One  criterion  that  can  be  applied  is  the  efficiency  of  transmission,  defined  by 

E '  ■  rii f  ■  °-233  23-» 
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Other  items  that  can  be  determined  are  the  equivocation, 

H(A|b)  =  H(A,B)  -  H(B)  =  1.628  bits, 
the  channel  matrix,  and  the  capacity. 

The  system  can  now  be  investigated  to  determine  the  value  of  troubleshooting 
strategies;  the  effects  of  regrouping  of  components  to  reduce  troubleshooting 
times;  and.  in  the  case  of  AIDS  type  systems,  the  effectiveness  of  the  system. 

3.8  ANALYTIC  MODELS 

Strictly  analytic  methods  are  another  means  of  optimizing  that  the  system 
analyst  can  employ.  One  such  technique  that  is  common  in  calculus  is  equating 
the  derivative  to  zero.  However,  two  other  techniques  are  worthy  of  mention: 
Lanchester's  equations,  and  the  calculus  of  variations. 

3.8  .1  Lanchester's  Equations 

Lanchester's  equations  deal  with  the  interactions  of  opposing  sides  in  a 
dynamic  battle.  In  their  simplest  form.  Lanchester's  equations  state  that  in  a 
multiple  engagement  "the  overall  effectiveness  of  a  force  equals  the  average 
effectiveness  of  the  individual  units  multiplied  by  the  square  of  the  number  of 
units  engaged.*  In  mathematical  terms  this  means  that 

H  ■  -rir 


dR 

dT 


-k2B 


where  B  and  R  are  the  numbers  of  blue  and  red  units,  respectively,  t  is  time,  and 
k1  and  k2  are  unit  effectiveness  factors. 

This  signifies  that  on  the  average  each  unit  will  in  a  given  time  score  a 
certain  number  of  effective  hits,  thereby  causing  the  number  of  units  killed  to 
be  directly  proportional  to  the  numericai  strength  of  the  opposing  force. 

These  equations  have  been  subsequently  modified  to  incorporate  other  factors 
affecting  force  strength,  such  as  production  rates. 


For  example,  one  such  modification  is  to  write  the  two  equations  as  being 
expressive  of  national  effectiveness  in  a  who^e  war.**  These  are  written  a8 
follows  : 


dR 

dt 


PR  - 


°LNR 


eBNB 


cTF  PB  °LNB  ”  eRNR 


*G .  Merrill, et.  al.,  "Operation  Research,  Armament.  Launching",  Principles  of 
Guided  Missile  Design.  L>.  Van  Nostrand  company,  inc.,  p  113. 

**Ibid. ,  p.  114. 
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where  P  =  production 

0^  =  operational  loss  percentage 

e  =  effectiveness  =  Number  of  enemy  destroyed - 

Number  of  friendlies  engaged 

N  =  number  of  forces  engaged 
B,R  =  blue  and  red,  respectively. 

The  equations  may  be  further  complicated  by  introducing  probabilities  into 
the  picture  and  by  exercising  them  through  a  simulation,  using  Monte -Carlo  tech¬ 
niques  to  determine  engagement  outcomes  under  more  realistic  conditions  and  with 
more  variables  introduced. 


To  illustrate  the  equations  simply,  consider  the  following  numerical 
examples.  Sides  A  and  B  have  10  units  each.  if  all  units  have  equal  effective¬ 
ness  per  unit  of  time,  the  engagement  will  be  a  draw.  Let  the  effectiveness 
(kills  per  unit  of  time)  of  A  be  0.1  and  B  0.2. 

Thus,  H  =  0.2B  AQ  =  10 


dB  =  0 ,1A 

IT  ' 


Bq  = 


The  resulting  time  histories  for  the  two  sides  are  approximately  as  follows: 


Time  Units 


Thus  in  this  simple  example  B  wins,  losing  but  three  units  while  all  ten  of  the 
opposing  A  are  lost,  and  yet  its  unit  effectiveness  was  only  twice  as  good  as 
A' s.  If  B1 s  effectiveness  were  raised  to  0.4  while  A's  remained  the  same,  A 
would  lose  all  ten  again,  but  B  would  lose  only  2,  Furthermore,  the  engagement 
would  require  three  time  units  whereas  before  it  required  seven. 
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From  the  example,  the  trade-offs  between  numbers  and  unit  effectiveness  (and 
time)  are  apparent.  Thus,  by  use  of  these  equations,  optimal  strategies  could  be 
devised  . 

3.8.2  Calculus  of  Variations 

The  calculus  of  variations  is  an  analytic  method  for  dealing  with  problems 
of  maxima  and  minima.  In  particular,  it  deals  with  finding  the  extrema  of  inte¬ 
grals  of  one  or  more  unknown  functions.  Thus,  in  the  calculus  of  variations,  the 
type  of  problem  that  is  addressed  is  as  follows:  find  F  such  that 

f  t2 

/  F(x,y,z,  x',  y',  z  ,  t)  dt  =  Maximum  or  Minimum 


The  mathematics  involved  In  solving  a  variational  problem  is  quite  complex; 
however,  electrical  or  mechanical  engineers  have  probably  encountered  problems 
that  could  be  solved  by  the  calculus  of  variations  or  have  applied  results  that 
were  derived  from  the  application  of  the  calculus  of  variations.  For  example, 
the  derivation  of  the  optimum  filter  is  a  direct  application.  jn  this  case,  a 
true  signal,  Y(t),  and  a  received  signal,  y(t),  (at  the  filter  output)  are  given, 
and  it  is  desired  to  find  the  filter  transfer  functions  such  that 


minimum 


A  second  example  of  the  use  of  this  technique  is  the  derivation  of  the 
equations  of  motion  of  vibrating  membranes,  plates,  etc.,  from  an  energy  stand¬ 
point  (e.g.,  using  Hamilton's  Principle).  in  this  case  the  problem  is  formulated 

t2r 


as 


\jC 

tl 


U(x,y,z,t)  -  K(x,y,z,t)  -  A(x,y,z,t) 


dt  =  Minimum 


where  U,  K,  A,  respectively,  represent  the  potential,  kinetic,  and  applied 
energies  in  the  system. 


The  general  approach  to  the  solution  of  these  problems  is  to  consider  the 
functions  fixed  at  their  end  points  but  free  to  vary  small  amounts  along  the  paths 
of  integration. 

3.9  DECISION  THEORY 

Decision  theory  represents  one  of  the  most  recent  developments  in  operations 
research.  It  has  found  considerable  prior  application  as  part  of  communications, 
radar,  and  pattern- recognition  systems.  However,  Its  applications  to  the  actual 
decision  process  have  been  relatively  recent  and  few. 

In  decision  theory  two  factors  contribute  to  a  decision: 

(1)  The  probability  of  the  outcomes  if  a  given  decision  is  made 

(2)  The  value  of  the  outcomes 
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The  theory,  then,  attempts  to  define  the  decision  process  in  terms  of  a 
number  of  states,  values  being  associated  with  each.  Application  of  the  theory 
results  in  identifying  a  best  course  of  action,  generating  alternative  states, 
establishing  new  values,  and  providing  a  dynamic  framework  for  the  decision 
process. 

3.10  COST-ESTIMATING  RELATIONSHIPS  AND  CONFIDENCE  INTERVALS 
3.10.1  Cost -Estimating  Relationships 

When  the  system  is  complex,  there  is  usually  no  simple  two-parameter  formula 
that  relates  system  characteristics  to  system  performance  or  system  cost  .  How¬ 
ever,  through  the  use  of  the  statistical  technique  known  as  linear  regression, 
the  equations  or  relationships  of  interest  can  be  approximated  by  a  straight  line 
or  by  a  hyperplane  —  that  is,  a  straight  line  in  n  dimensions. 

The  first  step  in  the  procedure  is  to  establish  a  list  of  system  parameters 
on  the  basis  of  engineering  judgment;  this  list  includes  system  characteristics 
that  are  expected  to  contribute  significantly  to  the  variability  of  system  per¬ 
formance  or  system  cost  at  any  time  during  the  research  and  development,  initial 
investment,  or  operating  phases  of  the  system's  life  cycle.  For  example,  if  the 
system  were  a  radio  receiver,  the  list  of  system  parameters  would  include  such 
items  as  weight,  volume,  sensitivity,  selectivity,  signal-noise  ratio,  stress 
characteristics,  and  cost  . 

Great  care  should  be  taken  in  compiling  these  lists,  because  the  case  of 
computation  and  the  adequacy  of  the  resulting  prediction  depend  primarily  on  the 
discrimination  exercised  at  this  point.  The  following  are  the  usual  priorities 
for  parameter  selection  in  the  regression  analyses: 

(1)  Parameters  that  are  considered,  on  a  cost/engineering  basis,  to  have 
a  second-order  effect  on  the  applicable  cost  category  are  excluded 
initially. 

(2)  Parameters  that  exhibit  little  variation  among  the  systems  in  the 
study  are  excluded  initially. 

(3)  Parameters  that  might  be  difficult  to  quantify  during  the  initial 
procurement  stages  have  lower  priority  than  others. 

(4)  Parameters  that  are  highly  correlated  with  one  or  more  other  para¬ 
meters  have  lower  priority. 

(5)  Parameters  are  selected  so  that,  if  possible,  at  least  one  from  each 
of  the  following  categories  is  initially  investigated: 

•  Mission  characteristics 
.  State-of-the-art  characteristics 
.  Complexity  or  quantity  characteristics 
.  Effectiveness  characteristics 
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One  parameter  may  represent  two  of  the  above  characteristics. 

For  example,  a  complexity  characteristic  such  as  number  of  active 
elements  will  often  correlate  well  with  reliability. 

( 6 )  Where  there  is  a  choice  between  two  or  more  equally  important 
correlated  parameters,  the  parameter  or  parameter  combination 
that  is  most  conducive  to  establishing  trade-off  relationships 
is  selected  initially. 

The  second  step  in  the  procedure  is  to  determine  the  coefficients  of  the 
regression  equation.  The  regression  equation  in  Section  2-12  can  be  written  as 

X1  =  A  +  BX2  +  CX3  +  DX4 

where  A,  B,  C,  and  D  are  coefficients,  or  constants,  that  have  to  be  determined 
and  the  X's  are  the  system  parameters  or  combination  of  system  parameters .  The 
constants  can  be  determined  by  solving  the  following  set  of  equations  simultane¬ 
ously: 

2X1  =  NA  +  B2X2  +  CSX3  +  DZX^ 

SXjXg  =  AzX2  +  BSX22  +  CZX2X3  +  DSXgX^ 

SX1X3  =  AsX3  +  BZXgXg  +  C2X2 2  +  DSX^ 

2XxX4  =  AsX4  +  BsX2X4  +  C2X3X4  +  D2X42 

where  N  is  the  number  of  samples. 

Once  the  constants  A,  B,  C,  and  D  have  been  found,  the  regression  equation 
is  determined,  and  the  value  of  the  multiple  correlation  coefficient,  r,  can  be 
calculated  from  the  following  formula: 

r  =  B(N  X^-SXjZXg)  +  CfNSX^Xj  -  SXjZx^ )  +  D(N2X-jX^  -  2x^X4) 

The  value  of  the  multiple  correlation  coefficient,  r,  will  always  be  between 
0  and  1,  and  if  the  value  of  r  is  1,  then  all  the  sample  points  lie  on  the  plane 
(or  hyperplane)  .  If  the  value  of  r  is  small,  say  less  than  0.7,  then  the  sample 
points  are  not  approximated  by  a  plane,  and  thus  some  other  form  of  a  prediction 
equation  should  be  tried. 

3.10.2  Confidence  Intervals 

The  next  area  of  interest  in  the  regression  technique  is  the  determination 
of  the  standard  error  of  the  estimate  and  the  confidence  intervals.  The  vertical 
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(X,)  3catter  of  the  sample  points  about  the  plane  is  measured  by  the  standard 
error  of  the  estimate,  S: 


S  = 


/  N-I 
V  N-K-l 


(1-r2) 


where 


fNXX-j2  -  (2X.,.)2 


N  (N-l) 


K  =  Number  of  variables  used  in  predicting  X^ 

N  =  Number  of  sample  points 
r  =  Correlation  coefficient 

Approximately  68  percent  of  the  sample  points  lie  within  ±S  of  the  plane 
determined  by  the  regression  equation,  and  95  percent  of  the  sample  points  lie 
within  ±23  (measured  in  the  X^  direction) . 

The  standard  error  of  the  estimate  gives  an  indication  of  the  spread  of  the 
original  data  points  about  the  regression  plane,  However,  when  the  regression 
equation  I  e  used  to  predict  the  cost  of  a  new  piece  of  equipment,  the  prediction 
Interval  or  confidence  interval  is  given  by 

— 

1  *h',Y  °u  <*i  -  V  <l)  -  V 

where 

X^  i s  the  predicted  value  of  X^  obtained  from  the  regression  equation 

S  is  the  standard  estimate  of  the  error 

t  at  (a/2,  n-2)  is  a  value  obtained  from  the  t  distribution  tables 

A 

X  IS  the  value  of  the  parameters  for  the  new  equipment 
X  is  the  mean  or  average  of  the  x's 

K  is  the  number  of  parameters  used  in  the  prediction  equation 
is  as  explained  below 


To  obtain  the  values  the  following  procedure  is  used.  if  the  para¬ 

meters  used  in  predicting  X^  are  X %,  and  X^,  then  the  first  step  is  to 
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calculate  the  quantities  V,  using  the  following  equations: 

V22  =  N2X22  -  (ZX2)2 

V33  =  N2X32  -  (2X3)2 

V44  =  NSX42  -  (SX4)2 

V23  =  V32  =  NSX2X2  -  2X22X2 

V24  =  V42  =  N2X2X4  "  2Xg2X^ 

V34  =  V43  =  N2X3X4  -  2X3BC4 

Now,  to  determine  the  values  for  U22,  U23,  anc*  ^24'  values  of  V  are 

substituted  in  the  following  equations,  which  are  solved  simultaneously: 

V22  u22  +  V23  U23  +  V24  U24  =  1 

v32  U22  +  V33  U23  +  V34  U24  =  0 

v42  u22  +  v43  U23  +  v44  u24  =  0 

To  determine  the  values  for  U32'  ^33'  and  ^34'  the  values  of  V  are  sub¬ 
stituted  in  the  following  equations,  which  again  are  solved  simultaneously: 

V22  u32  +  v23  u23  +  v24  u34  =  0 

V32  U32  +  V33  U33  +  V34  U34  =  1 

v42  U32  -  v43  U33  -  v44  U34  =  0 

Finally,  to  determine  the  values  for  U42,  U43,  and  U44,  the  same  values  of 
V  are  substituted  into  the  following  equations,  which  are  solved  simultaneously: 

^22  U42  ^23  ^43  V24  U44  =  ® 

^32  U42  +  ^33  ^43  +  ^34  u44  =  ® 

v42  U42  +  v43  U43  +  v44  u44  =  1 

The  confidence  intervals  can  now  be  found.  For  example,  if  the  95  percent 
confidence  interval  is  desired,  the  values  of  S,  N,  X,  and  U  are  substituted  in 
the  confidence  interval  equation,  along  with  the  value  of  t  (based  on  a  95  per¬ 
cent  confidence  level  and  the  number  of  samples,  N) .  The  relevant  statement  that 
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can  now  be  made  is  that  the  estimated  cost  of  the  new  equipment  is  X^,  and  there 
is  a  95  percent  probability  that  the  cost  will  be  somewhere  between  the  upper  and 
lower  confidence  limits. 

3.10.3  Example: _ Simple  Linear  Regression  (Two  Variables! 


Given  the  information  in  the  following  table,  determine  the  cost  for  a  new 
piece  of  similar-type  equipment,  the  xyz-2,  which  has  a  volume  of  30  cubic  feet. 


Existing 

Equipment 

Volume 
(Cubic  Ft.) 

cost 

(Dollars) 

URC-32 

20 

10,392 

WRT-2 

34 

12,278 

R-390 

2 

1,096 

URC-35 

3 

6,628 

URC-9 

12 

3,307 

SRC-21 

14 

4,366 

SRC-20 

18 

7,688 

URT-1 

36 

14,580 

XYZ-2 

30 

X1 

The  procedure  to  be  followed  consists  initially  of  linear  regression,  using 
the  method  of  least  squares.  If  it  is  found  that  the  data  are  not  essentially 
linear,  then  other  methods  are  tried,  such  as  logarithmic,  quadratic,  etc.,  until 
an  appropriate  prediction  equation  can  be  obtained. 

The  form  of  the  linear  equation  is 

X1  =  A  +  BX2 

where  X^  represents  the  cost  and  X2  represents  the  volume. 

The  values  of  the  coefficients  A  and  B  can  be  found  by  simultaneously  solving 
the  following  equations.  Since  there  are  only  two  variables,  the  relevant  equations 
(excluding  any  terms  containing  X^  or  X^)  are 

SXX  =  NA  +  B2X2 

SX^Xg  =  A2X2  +  B2X?2 
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From  the  information  in  the  preceding  table, 

SXX  =  10,392  +  12,278  +  ...  =  60,335 
zx2  =  20  +  34  +  2  +  .. .  =139 
SX^Xg  =  20(10,392)  +  34(12,278)  +...  =  1,411,620 
SX22  =  (20)2  +  (34)2  +...  =  3529 
N  =  8 

Substituting  these  values  into  the  above  equations  gives 

60,335  =  8A  +  B(139) 

1,411,620  =  A(139)  +  B(3529) 

Solving  these  equations  simultaneously  for  A  and  B  yields 

A  =  1840,  B  =  326  and 
X-,^  =  1840  +  326  X2 

Thus  the  cost  for  an  equipment  with  a  volume  of  30  is 
Xx  =  1840  +  326(30)  =  $11,600 

An  alternate  method  of  solution,  which  determines  the  value  of  the  corre¬ 
lation  coefficient  r  before  solving  for  A  and  B,  does  not  require  the  simultan¬ 
eous  solution  of  two  equations. 

The  correlation  coefficient,  r,  for  two  variables  is  given  by 


where 

the  standard  deviation.  crx,  of  a  number  of  samples,  N,  is 


the  mean  or  average  of  the  N  sample  points.  X,  i  s 

X  =  SC 

N 
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Note:  the  absolute  value  of  the  correlation  coefficient  will  be  somewhere 

between  land  zero.  It  has  been  found  empirically  that  an  absolute  value  of 
r  greater  than  0.7  will  yield  an  acceptable  result;  i.e.,  the  data .points  are 
essentially  linear.  If  the  data  points  actually  do  lie  on  a  straight  line,  the 
absolute  value  of  r  w  i  1  1  be  1.  A  negative  value  for  r  indicates  a  line  with  a 
negative  slope  . 

The  values  of  A  and  B  can  now  be  calculated  by  using  the  following  equations 

r  o_ 


B 


*2 


. A=  Y1  -  B  Tg 


For  this  example,  the  information  from  the  table  is  substituted  into  the 
above  expressions  to  yield  the  following: 

2X1  .  10,392  +  12,278  +  ...  =  60,335 

S(X1)2  =  (10,392) 2  +  (12,278)2  +  ...  =  605,554.500 

X1  =  60 ^  -  7,542 
o 


C  _  / 8(605, 55 11  ' 

1  8(8-1) 


C.r\  nnn 


zx2  =  20  +  34  +  2  +  ...  =  139 
s(x2)2  =  (20)2  +  (34)2  +  (2)2  + 
T2  -  If-  =  17 .4 


=  4,637 


3529 


/  8  (35on^  fim  ^ 
8(8-1) 


12.6 


r  =  [(20) (10, 392)  +  (34)(12,278)  +...]  -  8(7,542)(17.4)  =  0<q9 

8(4,637X12.6) 

B  =  (0. 89) (4637 )  _  326 

12.6 

A  =  7542  -  326  (17.4)  =  1840 

Therefore  X1  =  1840  +  326  X2  and  as  before,  for  an  equipment  with  a  volume  of  30, 


X1  =  1840  +  326  (30)  *  $11,600 
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Determine  the  standard  error  of  estimate  for  the  above  data,  and  the  95  per¬ 
cent  confidence  interval  for  the  xyz-2  equipment. 

The  vertical  scatter  of  the  data  points  about  the  regression  line 
X,  =  1840  +  326  Xc  is  measured  in  terms  of  the  standard  error  of  estimate,  S 

a.  d. 

where 


S  =  oY  1  - 

X1  \J 


It  has  been  determined  previously  that 


=  4637,  r  =  0.89 


therefore  S  =  4637  Jl  -  (0.8g)2  =  2130 


Note:  Approximately  68  percent  of  the  data  points  lie  within  ±  S  of  the 
regression  line,  and  95  percent  lie  within  ±  2S.  Therefore,  if  a  graphical  plot 
of  the  data  points  is  made,  the  parallel  lines  at  a  vertical  distance  of  2130 
from  the  regression  line  X-^  =  1840  +  326  x2  contain  approximately  68  percent  of 
the  data  points  and  the  parallel  lines  at  a  vertical  distance  of  4260  from  the 
regression  line  contain  approximately  95  percent  of  the  data  points.  As  the 
sample  size  increases,  the  number  of  data  points  within  ±  §  ancj  ±  2s  would  become 
closer  to  68  percent  and  95  percent,  respectively. 


To  calculate  the  confidence  interval  for  the  cost  of  the  xyz-2  equipment, 
the  following  equation  is  used:* 


*1  * 


4.1  (Xg 

J  "  "  I"-1)!  V2 


where 

t  i  s  a  value  obtained  from  the  t  distribution  tables 


X2  is  the  ordinate  of  the  regression  line  for  which  the  confidence 
interval  is  to  be  found 


Other  symbols  are  as  in  the  previous  example. 


*Note  -  This  is  another  form  of  the  equation  developed  previously  and  is  also 
applicable  for  simple  regression. 
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For  the  xyz-2  equipment, 

Xx  =  11,600 

S  =  2130 

t  =  (0.025,6)  =  2.45  (  from  t  table  for  95  percent  confidence  and 

8  data  points) 

Xg  =  30  cubic  ft 

X,  =  17.4 

=  12.6 

N  =  8 

The  95-percent  confidence  interval  is 


11,600  .  2.447  (  21301 

=  11, 600  ±  5900 

Therefore,  the  estimated  cost  of  the  xyz-2  equipment  is  $11,600,  and  there  is  a 
95  percent  confidence  that  the  cost  will  be  somewhere  between  $17,500  and  $5,700. 

This  wide  range  for  the  confidence  interval  is  quite  large.  However,  the 
significant  fact  is  that  without  an  indication  of  the  range  of  probable  values, 
the  decision-maker  would  have  no  feeling  for  the  accuracy  of  any  predicted  para¬ 
meter.  It  is  better,  of  course,  to  have  a  narrow  range  for  the  95-percent  con¬ 
fidence  interval,  but  this  can  be  achieved  only  if  additional  supporting  data 
are  available. 

3.11  EXPERIENCE  CURVES 

There  are  several  factors  that  can  reduce  the  unit  cost  of  an  equipment  as 
the  total  number  of  equipments  purchased  is  increased.  Two  such  factors  are  the 
initial  tooling  cost,  which  can  be  spread  out  over  a  larger  number  of  equipments; 
and  the  cost  of  materials,  which  can  be  reduced  for  a  quantity  purchase. 

Another  factor  that  can  reduce  the  unit  cost  of  an  equipment  (unrelated  to 
the  two  factors  above)  is  the  learning  curve,  or  experience  curve  Y  =  aXb ,  where 

Y  =  cost  to  manufacture  equipment  X 
a  =  cost  to  manufacture  equipment  number  1 
X  =  equipment  number 

b  «=  exponent  of  experience-curve  slope 
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Empirical  data  have  shown  that  the  experience  curve  is  appropriate  for 
predicting  the  costs  of  aircraft  engines  and  airframes  and  several  types  of 
electronic  equipments.  Normally,  the  experience  curves  are  developed  by  the 
equipment  manufacturer.  The  experience  curve  is  based  on  the  fact  that  as  the 
quantity  of  equipments  being  manufactured  is  doubled,  the  cost  to  manufacture 
each  successive  equipment  is  reduced  by  a  constant  percentage.  If  one  equip¬ 
ment  costs  $1000  to  manufacture,  the  second  equipment  is  following  a  90-percent 
learning  curve,  and  the  eighth,  sixteenth,  and  thirty- second  equipment  each  cost 
90-percent  less  than  the  previous  quantity. 

Normally,  the  cost  reduction  described  by  the  experience  curve  is  due 
entirely  to  the  reduction  in  man-hours  necessary  to  produce  an  equipment,  through 
the  natural  process  of  man  learning  his  job  better  by  repetition.  There  are, 
however,  fully  automated  production  lines  with  experience  curves  based  on  the 
fact  that  as  the  production  line  is  operated,  supervisors  can  develop  improve¬ 
ments  and  short-cuts  in  the  process. 

The  mathematical  method  for  fitting  data  to  the  experience  curve  Y  =  aXb 

is  log-log  least- squares  regression.  Once  the  constants  "a"  and  "b"  have  been 
found  by  using  the  regression  technique,  the  unit  cost  of  any  equipment  can  now 
be  found.  For  example,  suppose,  "a",  the  cost  of  the  first  equipment,  is 
$20,000,  and  the  value  of  "b"  is  -0.322*.  The  cost  for  the  sixteenth  equipment 
i s  given  by 

0  322 

Yl6  =  (20,000)  (16)-  =  $8,200 

Tables  have  been  developed  by  several  Army  agencies  that  can  be  used  to 
reduce  the  amount  of  calculation  for  unit,  average,  and  cumulative  costs  for  any 
number  of  equipments  with  any  slope. 

Another  method  that  is  usually  a  good  approximation  for  determining  the 
experience  curve  is  the  "eyeball"  method,  i.e.,  plotting  the  data  points  and  then 
drawing  a  straight  line  through  the  spread  of  points  with  a  straight  edge.  Graphi¬ 
cally,  the  data  points  are  plotted  on  log.-log  paper,  and  if  the  relation  Y  =  aXb 

exists,  then  the  data  points  will  fall  essentially  along  a  straight  line.  The 
slope  of  this  experience  curve  can  be  found  readily  from  any  two  points  on  the 
straight  line  whose  ordinates  are  separated  by  a  factor  of  two  (one  equipment 
quantity  double  the  other).  For  example,  if  it  is  found  from  the  curve  the  ninth 
equipment  costs  $500  and  the  eighteenth  equipment  costs  $450,  then  the  slope  of 
the  experience  curve  i s  90  percent.  To  establish  the  values  of  "a"  and  "b"  (in 
the  equation  Y  =  axb)  from  the  graph,  note  that  "a"  represents  the  cost  of  the 


*  The  value  of  b  will  always  be  negative  unless  the  unit  cost  for  succeeding 
equipments  increases. 
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first  equipment,  and  that  "b"  can  be  determined  from  the  slope.  The  slope  of 
the  experience  curve  and  the  exponent  b  are  related  by  the  formula 

%  slope  =  (2)b  x  100$ 

which  follows  logically  from  the  observation  that  the  cost  of  the  first  equip¬ 
ment  (for  X  =  l)  is  equal  to  "a"  and,  by  definition,  the  cost  of  the  second  equip- 
ment  is  equal  to  "a"  times  the  slope;  therefore  2°  (forX  =  2)  must  be  equal  to 
the  slope. 

3.12  COST-SENSITIVITY  ANALYSIS 

3  .12.1  General 

There  are  two  main  areas  of  usage  in  which  cost-sensitivity  analysis  is 
used.  First,  the  individual  cost  constituents  should  be  checked  to  determine 
how  changes  in  them  affect  the  total  cost.  For  example,  a  5-percent  change  in 
the  maintenance  cost  of  an  avionic  system  may  change  the  total  lifetime  cost  of 
the  system  by  15  percent,  while  a  10-percent  change  in  the  equipment  cost  changes 
the  total  lifetime  cost  by  less  than  1  percent.  The  implication  here  is  that  the 
cost  analyst  should  concentrate  on  refining  the  maintenance-cost  prediction, 
whereas  a  relatively  gross  estimate  of  the  equipment  cost  will  be  sufficient. 

Secondly,  any  assumptions  that  were  made  in  the  analysis  should  be  checked 
to  determine  how  changes  in  the  assumptions  affect  the  total  cost.  For  example, 
if  it  was  assumed  that  the  equipment  would  be  operated  for  100  hours  per  month, 
the  total  costs  should  be  calculated  for  operating  times  of  say  75  and  125  hours 
per  month  to  determine  the  effect  of  this  assumption  on  the  overall  cost.  Of 
course,  if  the  total  cost  is  sensitive  to  any  assumption,  the  results  of  the 
sensitivity  analysis  should  be  shown  to  the  decision-maker  with  the  range  of  the 
assumed  value  indicated. 

3.12.2  Cost-Sensitivity-Analysis  Problem 

The  cost  information  given  below  for  two  alternative  communications  systems 


Cost  Categories 

Total  System  Cost 
(10-year  lifetime) 

Communication 
System  A 

Communication 
System  B 

R&D 

$  10,000,000 

$  12,000,000 

Equipment  Acquisition 

300,000,  000 

320,000,000 

Spares  and  Spare  Parts 

14,000,000 

8,000,000 

Initial  Maintenance 
Facilities 

4,500,  000 

6, 000, 000 

Publications 

500,  000 

800, 000 

Maintenance 

160,000,000 

120,000,000 

Annual  Training 

1,000,000 

goo, ooo 

Annual  Facilities 

2,000,000 

2,500,000 
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The  cost  figures  represent  average  (or  expected)  values,  with  some  standard  error 
of  estimate.  One  cost-estimating  relationship  was  developed  for  each  category; 
therefore,  the  standard  error  for  each  category  is  the  same  for  system  A  and  B. 
but  the  categories  will  generally  have  different  standard  errors.  In  other  words, 
the  standard  errors  are  identical  horizontally,  but  not  necessarily  vertically. 

Question  1:  In  which  cost  category  does  cost  uncertainty  have  the  greatest 
impact  on  the  cost  comparison,  and  how  great  is  this  impact? 

Question  2:  Are  the  results  sensitive  to  the  assumption  that  the  system  will 
be  in  operation  for  10  years? 


Answers 


Answer  to  Question  1 

The  cost  totals  for  each  system  are: 

A:  $492,000,000 

B:  $470,200,000 

A,  then,  is  ostensibly  more  expensive  than  B.  However,  because  of  the 
existence  of  standard  errors,  a  cost- inversion  may  be  possible,  so  that  B,  in 
fact,  is  the  more  expensive  system. 

Equipment  Acquisition  and  Maintenance  are  the  two  biggest  categories; 
between  them  they  account  for  more  than  90  percent  of  the  total  system  cost. 

Assume  that  if  a  cost- inversion  exists,  it  is  due  either  to  Acquisition  or 
Maintenance  cost  errors  (or  both),  the  other  categories  being  so  small  relatively 
(in  terms  of  money)  that  they  are  essentially  constants. 

The  difference  between  system  costs  is  $21,800,000.  Thus  if  at  least  one- 
half  of  $21,800,000,  or  $10,900,000  were  simultaneously  added  to  the  total  cost 
of  B  and  subtracted  from  the  total  cost  of  A,  an  inversion  would  result. 

Now,  the  total  cost  for  system  A  can  be  written  as 

CostA  =  32,000,000  +  (EA  +  MA) 

where 

Ea  =  Equipment  Acquisition  Cost 
Ma  =  Maintenance  Cost 

The  total  cost  for  system  B  can  be  written  as 

Costg  =  30,200,000  +  (Eg  +  Mg) 
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The  probability  that  system  B  costs  more  than  system  A  is  given  by 


P  (cOStg  >  COStA)  =  p 


30,200,000  +  (Eb  +  Mg)  >  32,000,000  +  (Ea  +  MA) 


*=  P 


(Eg  +  Mg)  -  (Ea  +  Ma)  >  1,800,000 


=  P 


Z>  1,800,000 


Since  the  tabulated  values  of  EA,  Eg,  MA,  and  Mg  were  obtained  by  regression 
techniques,  these  values  are  the  expected  values. 


Thus  if  Z  is  the  random  variable 


z  =  (Eg  +  Mg)  -  (Ea  +  Ma) 


then  the  expected  value  of  Z  i  s  given  by 
Z  =  (Eg  +  Mg)  -  (Ea  +  Ma)  =  (320,000,000  +  120,000,000)- 

+  (160,000,000)  j 

or  2  =  -20,000,000 


(300,000,000) 


Ideally,  the  standard  errors  of  estimate  for  EA,  Eg,  Ma,  and  Mg  (and  the  other 
costs  as  well)  should  have  been  furnished.  Since  they  have  not,  a  "worst"  case 
estimate  is  obtained  as  follows: 


Assume  that  the  percentage  errors  in  E  and  M  do  not  exceed  some  nominal 
value,  say  10  percent;  thus  10  percent  of  EA  =30,000,000,  and  10  percent  of 
Eg  =  32,000,000.  If  these  values  arc  considered  to  represent  three  standard 
errors,  which  almost  guarantees  that  the  error  will  be  less  than  10  percent, 
then 

Sr,  =  10,000,000  and  Sp  =  10,700,000 
jit 


However,  from  the  assumption  that  the  same  regression  equation  was  used, 

must  equal  Sg  ,  so  that  if  ^  and  ^g  are  "averaged", 

B 


Similarly, 


S_  =  S-,  =  10,300,000 

ea 


\  -  \  = 4’700’000 


Then  SZ  =  yj4A  +  4B+  SMA  +  ^  =  V[2  (10 . 3) 2  +  2(4. 7)  2]  X  1012 

=  V256.36  X  106 
=  16.000.000 
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The  problem  is  shown  graphically  below.  The  curve  represents  the  distribution 
of  equipment  acquisition  and  maintenance  costs:  i.e.,  the  equipment  acquisition 
and  maintenance  cost  of  system  B  is  "expected"  to  be  $20,000,000  less  than  the 
equipment  acquisition  and  maintenance  cost  of  system  A.  and  the  standard  error 
of  this  distribution  is  $16,000,000.  The  zero  line  represents  that  point  at 
which  the  equipment  acquisition  and  maintenance  costs  for  A  and  B  are  equal.  The 
$1,800,000  line  represents  that  point  at  which  the  total  system  costs  for  A  and 
B  will  be  equal. 


The  probability  that  the  cost  lies  to  the  right  of  the  $1,800,000  line  (a  cost- 
inversion)  is  given  by 


h 


1 


-P- 


a 

where  h1  is  the  value  of  interest  on  the  normal  density  function 
u  is  the  mean  of  the  function 
a  is  the  standard  deviation 


Substituting  the  values  from  the  example  problem  into  the  above  equations 
yields 


1,800,000  -  (-20,000,000)  =  1^6 

16,000,000 


which,  from  Table  C-l  in  Appendix  C,  corresponds  to  0.0869,  , orapproximately 

9  percent.  Therefore,  for  an  estimated  maximum  error  of  10  percent  in  the  acqui¬ 
sition  and  maintenance  cost,  the  probability  that  system  B  is  really  more  expen¬ 
sive  than  system  A  is  9  percent.  if  this  process  is  repeated  for  an  estimated 
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maximum  error  of  100  percent,  the  calculations  lead  to  the  result  that  the 
probability  that  system  Bis  really  more  expensive  than  system  A  is  33  percent. 

The  answer  to  question  1,  then,  is  that  uncertainty  in  the  predictions  of 
equipment  acquisition  and  maintenance  cost  have  the  greatest  impact  on  the  cost 
comparison;  and  if  it  is  assumed  that  these  predictions  may  be  off  by  as  much  as 
100  percent,  there  is  still  only  a  33-percent  chance  that  the  cost  of  system  B 
will  be  more  than  the  cost  of  system  A. 

Answer  to  One  si  ion  2 

For  this  example,  assume  that  Maintenance,  Training,  and  Facilities  are 
time-based  costs,  and  assume  further  that  they  are  linear  with  time  (although 
any  function  other  than  linear  could  also  be  handled  easily  with,  for  example, 
a  graphical  solution). 

Then  costs  would  be  categorized  as  follows: 

.c-as-Ls.  System  A 

Fixed  Costs  $329,000,000 

Annual  Costs  16,300,000 

(per  year) 

For  y  years,  the  cost  of  system  A,  therefore,  will  be 
CostA  =  329,000,000  +  16,300,000  y 

Similarly, 

Costg  =  346,800,000  +  12,340,000  y 


System  B 
$346,800,000 
12,340,000 


Thus  CostA  s  CostB  when 

329,000,000  +  16,300,000  y  3  346.800,000  +  12,340,000  y, 
or  when  ys~  4.5  years 

The  answer  to  Question  2,  then,  is  that  the  results  are  not  sensitive  to 
the  assumption  that  the  system  will  be  operated  for  10  years  —  that  is,  the 
cost  of  system  B  will  be  lower  than  the  cost  of  system  A,  unless  the  system  is 
to  be  in  operation  for  less  than  4.5  years. 

(Note  very  carefully  the  assumption  of  linearity,  i.e.,  that  the  cost  is 
directly  proportional  to  time.  This  assumption,  for  specific  systems,  may  very 
well  not  be  true.) 
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CHAPTER  4 

BASIC  MATHEMATICAL  AND  STATISTICAL  CONCEPTS* 


4.1  INTRODUCTION 

Basic  mathematical  and  statistical  concepts  are  reviewed  in  this  chapter. 
Topics  include  algebraic  principles  and  formulas,  the  various  types  of  probabil¬ 
ity  distributions,  and  procedures  for  statistical  estimation. 

4.1.1  Preliminary  Definitions 

Some  of  the  principal  terms  used  in  this  discussion  are  defined  as  follows: 

Random  Outcome.  The  value  of  an  empirical  observation  that  cannot  be 
predicted  (lack  of  deterministic  regularity)  but  has  statistical  regularity  in 
that  the  value  has  a  relative  frequency  of  occurrence  in  a  series  of  independent 
observations  of  the  phenomenon  (the  result  of  tossing  a  die,  the  time-to-failure 
of  a  device,  etc.). 

Trial.  An  action  or  experiment  that  yields  a  random  outcome  (tossing 
a  die,  life-testing  a  device). 

Independent  Trials.  Trials  of  which  the  outcome  of  one  has  no  effect 
on  the  outcome  of  others  that  follow. 

Event.  A  set  of  outcomes.  The  event  has  occurred  if  one  of  the  out¬ 
comes  of  the  set  is  observed  on  a  trial.  (If  the  event  is  an  even  number  on 
the  toss  of  a  die,  it  occurs  if  the  number  2,  4,  or  6  is  observed.) 

Independent  Events.  Sets  of  outcomes  based  on  independent  trials. 

Mutually  Exclusive  Events.  Two  or  more  events  that  cannot  occur 
simultaneously  (odd  and  even  numbers  on  one  toss  of  a  die). 

4.1.2  Notation 

The  following  probability  notation  applies: 

P(E)  =  probability  of  event  E,  where  0  ^  P(E)  s  1 
P(E)  =  probability  of  event  "not  E"  =  1  -  P(E) 

*Some  of  the  material  used  in  this  chapter  was  developed  by  ARINC  Research 
Corporation  for  the  U.  S.  Weather  Bureau  under  Contract  Cwb-11349.  Reproduced 
by  permission  of  the  U.  S.  Weather  Bureau. 
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P(E-l  +  Eg)  =  probability  of  events  E1  or  Eg  (or  both  if  they  are  not 
mutually  exclusive) 


P ( E^Eg )  =  probability  of  both  events  E^.  and  Eg 

4.2  DEFINITIONS  OF  PROBABILITY 

4.2.1  Classical  Definition 


The  classical  definition  of  probability  is  as  follows: 

If  an  experiment  can  result  in  n  different,  equally  likely 
and  mutually  exclusive  outcomes,  and  if  r  of  these  outcomes 
correspond  to  event  E,  the  probability  of  E.  denoted  by  P(E), 
is  the  ratio 


P(E)  =  r/n  (1) 


Example:  If  a  card  is  drawn  at  random  from  a  full  deck, 
there  are  n  =  52  mutually  exclusive  and  equally  likely  out¬ 
comes;  r  =  4  of  these  are  the  event  of  drawing' a  king'.  The 
probability  of  drawing  a  king,  from  Equation  1,  is  4/52  =  1/13. 

The  classical  definition  of  probability  is  one  that  involves  an  a  priori 
evaluation  and  is  useful  only  if  all  the  possible  outcomes  can  be  enumerated  and 
are  equally  likely  and  mutually  exclusive.  "Equally  likely"  can  be  described 
as  the  lack  of  any  bias  favoring  one  outcome  over  another  in  a  trial. 

4.2.2  Relative- Frequency  Definition 

The  relative-frequency  definition  is  as  follows: 

The  probability  of  an  event  is  the  limiting  ratio  of  the 
number  of  outcomes  favorable  to  an  event  (r)  to  the  number 
of  trials  performed  (n)  as  the  number  of  trials  approaches 
infinity.  If  n  is  large,  the  ratio  r/n  can  be  used  to 
estimate  the  probability. 

Example:  In  the  life-testing  of  100  devices,  it  was 
observed  that  15  failures  occurred  before  20  hours.  The 
estimated  probability  of  failure  before  20  hours  is  there¬ 
fore  15/100  =  0.15.  'The  relative-frequency  definition  of 
probability  requires  a  statistical  estimation  involving 
valid  experiments  and  sufficient  data  to  yield  an  estimate 
that  is  fairly  stable. 


4.3  ALGEBRAIC  PRINCIPLES  AND  FORMULAS 

The  following  algebraic  principles  and  formulas  are  useful  for  applying  the 
classical  definition  of  probability  in  those  cases  in  which  it  is  valid. 

4.3.1  Two  Basic  Counting  Principles 

The  two  basic  counting  principles  are  as  follows: 

(1)  If  event  A  can  occur  in  "a"  ways,  event  B  can  occur  in  "b"  ways,  and 
both  can  occur  together  in  "c"  ways,  then  A  or  B  or  both  can  occur  in 
(a+  b-c)  ways. 
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Example  a;  Spade  or  heart  in  one  draw:  A  =  spade, 

B  =  heart .  5""=  13,  b  =  13,  c  =  0.  A  or  B  in  13  +  13  = 

26  ways. 

Example  b:  Spade  or  ace  in  one  draw:  A  =  spade, 

B  =  ace.  a  =  13,  b  =  4,  c  =  1.  A  or  B  in  13  +  4  -  1  = 

16  ways. 

(2)  If  there  are  "a"  ways  of  performing  the  first  operation  and  "b"  ways 
of  performing  the  second  operation,  given  that  the  first  operation 
has  occurred,  there  is  a  total  of  a  x  b  possible  ways  for  both 
operations. 

Example  a:  Throwing  an  even  number  on  a  die  and 
drawing  an  ace  from  a  deck:  a  =  3,  b  =  4.  Total  number 
of  ways  =  3  x  4  =  12, 

Example  b:  Drawing  two  spades  from  a  deck  of  cards: 
a  =  lT,  b  =  12.  Total  number  of  possible  ways  =  13  x  12  = 

156. 

4.3.2  Permutat ions 


A  permutation  is  a  particular  arrangement  of  a  collection  of  objects.  The 
total  number  of  permutations  of  n  different  objects  is 

P(n,  n)  =  n(n  -  l)  (n  -  2)  .  .  .  3  x  2  x  1  (2) 

=  n! 

The  total  number  of  permutations  of  n  objects  taken  k  at  a  time  is 

P(n,  k)  =  rn.^ik.),  (3) 

if  which  are  alike,  kg 

II  -I  - 

i=l 


The  total  number  of  permutations  of  n  objects,  k^ 
of  which  are  alike,  .  .  .  ,  k  of  which  are  alike 


(Note:  0!  is  defined  as  equal  to  1.) 

Example  a:  For  the  letters  ABCDE.  there  are  51  =  120 

5  1 

permutations.  There  are  ^ t  =  60  permutations  if  three 
of  the  five  letters  are  to  be  selected. 
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Example  b:  In  the  word  SUCCESS,  there  is  one  U  (k^  =  1), 
one  E  (k2  =  l),  two  C's  (k^  =  2),  and  three  S's  (k^  =  3). 

From  Equation  4,  the  total  number  of  possible  permutations  of 
all  seven  letters  in  the  word  SUCCESS  is 

7 •  =  420 

1 ! 1 ! 2 ! 3 1 


4.3.3  Combinations 

A  combination  is  the  nmnber  of  ways  in  which  k  out  of  n  different  items 
can  be  selected  without  regard  to  order,  symbolized  by 

(£)  C”  (5) 


where 


fn,i-  Si 
vk’l_  k!  (n-k) ! 


Example:  For  a  unit  composed  of  five  components,  at 

least  three  must  be  successful.  How  many  ways  can  the  unit 
be  successful? 


From  Equation  5, 


(3)  +  (4)  +  (5)  = 


=  10  +  5  +  1  =  16 


4.3.4  Basic  Probability  Laws 
4.3.4. 1  Addition  Law 

If  A  and  B  are  two  mutually  exclusive  events,  the  probability  that  either 
of  them  will  occur  in  a  single  trial  is  the  sum  of  their  respective  probabilities, 
o  r 


P( A  +  B)  =  P(A)  +  P(B)  (6) 

In  general,  if  there  are  k  mutually  exclusive  events, 

P(A1  +  A2  +  .  .  .  +  Ar)  =  P(A1)  +  P(Ag)  +  .  .  .  +  p(Ak)  (7) 

If  the  two  events  A  and  B  are  not  mutually  exclusive,  the  probability  that  at 
least  one  of  them  will  occur  is 

P( A  +  B)  =  P ( A)  +  P (B)  "  P (AB)  (8) 
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For  three  non-mutually  exclusive  events, 

P( A  +  B  +  C)  =  P( A)  +  P(B)  +  P(C)  -  P(AB)  -  P(AC)  -  P(BC)  +  P(ABC)  (9) 

The  most  general  form  of  the  addition  law  states  that  the  probability  of  an 
event  is  the  sum  of  its  mutually  exclusive  forms.  It  it  is  assumed  that  A  and 
B  are  not  mutually  exclusive  but  that  the  three  events  (A  and  B).  (A  and  not  B). 
and  (B  and  not  A)  are  mutually  exclusive,  then 

P(A  +  B)  =  P ( AB)  +  P(AB)  +  P(AB)  (10) 

4. 3. 4. 2  Multiplication  Lau 

If  events  A  and  B  are  independent,  the  probability  of  the  compormd  event  A 
and  B  is  equal  to  the  product  of  their  respective  probabilities,  or 

P(AB)  =  P ( A)  P(B)  (11) 

The  extension  to  more  than  two  events  follows  directly, 

4. 3. 4. 3  Conditional  Probability 

If  events  A  and  B  are  not  independent  __  i.e.,  the  occurrence  of  one  affects 
the  occurrence  of  the  other  --  then  conditional  probabilities  exist.  The  con¬ 
ditional  probability  of  A  given  that  B  has  occurred  is  denoted  by  P(A|B);  sim¬ 
ilarly,  the  probability  of  B  given  that  A  has  occurred  is  denoted  by  P(B|A).  if 
events  A  and  B  are  not  independent, 

P(AB)  =  P ( A)  P(B|A)  =  P (B)  P(A|B),  (12) 

which  reduces  to  P(A)  P(B)  if  A  and  B  are  independent. 

For  three  events. 


P ( ABC )  =  P(A)  P(B|A)  P(C|AB) 


(13) 


Also, 


p(a|  b)  = 


P(AB) 

pTbT 


(14) 


(15) 
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Equations  14  and  15  lead  to  a  form  of  Bayes1  Theorem. 

p/fll-ot  PfAl  P(BlA)  P ( A)  p(b|a) _ 

P(AIB'  P(b)  _  P  [A)  P(B|A)  +  P(E) P(B|ft) 

In  this  application,  P(A)  and  ?CK)  are  usually  a  priori  probabilities  of  the 
events  A  and  not  A.  It  is  necessary  to  modify  P(A)  on  the  basis  that  event  B 
has  occurred  in  some  experiment  whose  outcome  is  known  to  be  influenced  by  A, 
as  reflected  by  the  terms  P(BjA)  and  P(B|A). 

Example ;  Assume  that  a  box  of  100  outwardly  indistinguishable 
parts  Is  composed  as  follows: 


Manufacturer 

Quality 

A 

B 

C 

Totals 

Good,  G 

40 

27 

8 

75 

Bad,  (T 

10 

3 

12 

25 

Totals 

50 

30 

20 

100 

The  following  probabilities  are  based  on  the  classic  definition 


.  Total  number  of  outcomes  favorable  to  Event  E 
total  number  of  possible  outcomes 


and  on  the  probability  laws  discussed  above. 


Case  1 :  For  a  random  selection  from  the  box,  what  is  the  probability  of: 

(a)  Drawing  a  part  manufactured  by  A?  From  Equation  1, 


P{A) 


1 

2 


(b)  Drawing  a  bad  part? 


_  I§n  -  J 

(c)  Drawing  a  part  manufactured  by  C  which  is  also  bad? 

P(CG)  =  T5§  =  0.12 


(d)  Drawing  a  bad  part  manufactured  by  C,  given  that  a  part  manufactured 
by  C  was  selected? 


4-6 


AMCP  706-191 


Counting  indicates  that  there  are  a  total  of  20  possible  ways  for  selecting  a 
C  part,  and  in  12  cases  bad  parts  will  be  selected:  hence,  from  Equation  1, 

P(GjC)  =  ^  =  0.60 


Similarly,  from  Equation  14, 

P(G|C)  =  §t|§  =  0.60 

Case  2:  If  one  part  is  drawn  randomly.  Equation  6  yields 
P (A  +  B)  =  P(A)  +  P(B)  =  +  i$y  =  0.8  =  l-P(C)  =  1  -  iS&r  =  0.8 


From  Equation  8, 

P( A  +  E)  =  P( A)  +  P(G)  -  P(AG)  =  I  +  1  -  =  0.65 

3) 

Counting  indicates  that  the  number  (A  or  -  50  +  25  -  10  -  65.  The  number  of 
possible  outcomes  =  100. 


P(  A  +  <J)  =  ^  =  0.65 

Case  3:  If  two  draws  are  made,  what  is  the  probability  that  both  items 
selected  are  manufactured  by  A.  for  the  following: 

(a)  Drawing  with  replacement,  independent  events? 


P(A1Aa) 


Number  of  ways  of  drawing  two  items  manufactured  by  A 
l  otal  possible  number  of  two-ilcm  draws 


-  ^0  x  50 


-  100  X 


_  1/4 


or,  from  Equation  11, 

P(A1A2)  =  P(A1)  P(A2)  =  1/2  x  1/2  =  1/4 


(b)  Drawing  without  replacement,  dependent  events? 

P(A  »  )  =  50  x  49  _  _49 
v *1*2 '  100  x  99  “  19B 


or,  from  Equation  12, 

P(AlA2)  =P(A1)  P(Ag|Al)  =ix  4|=_4| 
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■Case  4:  What  is  the  probability  of  selecting  a  part  produced  by 
manufacturer  A  on  the  first  draw  and  a  bad  item  (i?)  on  the  second  draw? 

(a)  With  replacement: 

PU-jSg)  =  P(A1)  P(G2)  =  1/2  X  1/4  =  1/8  =  0.1250 


(b)  Without  replacement: 

*(*!&,>  =  p (Aj)  P(G2|A1) 

Since  (A  on  first  draw)  consists  of  the  events  A^G^  (A  and  G  on  first 
draw)  or  A/Zf,  (A  and  G  on  the  first  draw),  then 

PU^)  =  P(A1G1)  PtGgjA^)  +  PU^)  P^Ja^) 

=  ( 15§)  x  (®)  +  (iBs)  (H) 

=  0.1253 

Case  5  (Bayes  Theorem  Example):  Assume  that  the  parts  are  distributed  in 
two  identical  boxes  as  follows: 


Manufacturer  A 

G 

G 

40 

10 

Tffitlail 

50 

Not  Mfr 

A  (I) 

G 

35 

0 

15 

J2SSL 

58 

If  a  box  is  chosen  at  random  and  a  part  is  selected  from  the  box,  what  is  the 
probability  that  the  part  is  manufactured  by  A  if  it  is  found  to  be  good? 

The  a  priori  probability  of  choosing  a  part  manufactured  by  A  is  P(A)  = 
1/2;  that  of  choosing  apart  manufactured  by  not-A  is  ?(J.)  =  1/2,  since  the 
boxes  are  identical  and  one  part  is  randomly  chosen.  Front  Baves  Theorem 
(Equation  16),  the  probability  that  box  A  is  chosen,  given  that  a  good  item  is 
selected,  is 


P(A|G) 


FTO 


lx  40 


-T 40— P— 25-  ~ 

5U  2  x  5U 


2  X 


0.533 
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Hence,  by  Bayes  Theorem,  the  prior  probability  that  a  part 
manufactured  by  A  was  chosen  is  modified  from' 0.5  to  the 
posterior  probability  of  0.533  on  the  basis  of  the  information 
that  a  good  part  was  selected  from  the  chosen  box. 

4.3.5  Application  of  Probability  Laws  to  Reliability 

In  most  of  the  applications  of  probability  theory  to  reliability  work,  the 
events  in  question  are  expressed  in  terms  of  a  time  variable;  hence,  the  proba¬ 
bilities  themselves  are  not  constants  but  are  functions  of  time,  denoted  by  t. 

The  above  formulas  hold  equally  well  when  interpreted  as  functions  of  time. 

As  an  example,  the  reliability  at  time  t  is  equivalent  to  a  probability  of 
no  failure  before  t.  If  we  have  two  equipments,  a  and  b,  the  probability  that 
both  operate  is,  by  the  multiplication  law  (Equation  11)  and  under  the  assumption 
of  independence. 


Rab  <*>  =  «»(*>  ■ 

where  R^t)  is  the  reliability  at  time  t  of  equipment  i. 

If  there  are  two  equipments,  a  and  b,  and  if  the  system  is  successful  at 
time  t  if'either  a  or  b  or  both  are  operable,  then  by  the  addition  law  (Equation  8), 
the  system  reliability  R  (t)  is 

R3(t)  =  Rjt)  +  Rb(t)  -  Rab(t) 


4.4  PROBABILITY  DISTRIBUTIONS 

4.4.1  Definitions 

The  following  definitions  are  pertinent  to  a  discussion  of  probability 
distributions  or  densities: 

Random  Variable.  A  quantity,  x,  for  wb.ich--for  every  real  number  C-- 
there  exists  a  probability  that  x  is  less  than  or  equal  to  C. 

Discrete  Random  Variable.  A  random  variable  that  can  take  on  only  a 
finite  or  countable  number  of  distinct  values.  The  random  variable  "number  of 
failures  within  the  fixed  time  interval  [o,  t] "  is  discrete. 


Continuous  Random  Variable.  A  random  variable  that  can  take  on  any 
value  within  an  interval  (equivalent  to  taking  on  any  one  of  a  non-denumerable 
infinity  of  values).  The  random  variable  "time  to  failure"  is  continuous. 


ProbabiliD  Dcnsit\  Function.  A  mathematical  function,  say  f  (x) , 
which,  for  discrete  random  variables,  gives  the  probability  that  the  random  vari¬ 
able  equals  x.  For  continuous  random  variables.  f(x)  gives  the  probability  that 

rb 

x  lies  in  an  interval,  say  [a,  b  ],  from  the  equation  P  [a  <  x  §  b]  =  /  f(x)dx. 
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The  probability  that  a  continuous  random  variable  will  take  on  a  specific  value 
is  defined  to  be  zero  (e.g.,  the  probability  that  a  device  will  fail  at  exar.tK 
98.000  .  .  .  hours  equals  zero).  Intervals  rather  than  points  must  be  considered 
for  the  usual  continuous  cases. 

Cumulative  Distribution  Function.  The  mathematical  function  that 
expresses  the  probability  that  the  random  variable  x  is  less  than  or  equal  to 
some  value  as  determined  from  the  probability  density  function. 

If  X  represents  a  given  value  of  x,  then  for  a  discrete  random  variable  whose 
lower  limit  is  L,  the  cumulative  distribution  function,  P(X),  is 

X 

F(X)  =  £  f(x)  (17) 

x=L 


For  a  continuous  random  variable. 


F(X)  =  J  f(x) 


x)dx 


(18) 


4.4.2  Properties 

If  x  represents  the  random  variable,  and  f(x)  represents  the  probability 
density  function  of  x,  then  f(x)  has  the  following  properties: 

(1)  f(x)  2  0  for  all  x 


f(x)  =  1  *  1  -  if  x  is  discrete* 


(2) 


(19a) 


p  00 

/  f(x) 


dx  =  l  *  ■■  if  x  is  continuous** 


The  probability  that  x  will  take  on  a  value  in  the  interval  la>  ls  (19b) 

b 


P  [a  <  x  s  bj  = 


y. 

/rJaf(x)  if  x  is  di 


t=a 1  (x) 
b 


screte 


/; 


f(x)dx  if  x  is  continuous 


'I 

**x 


represents  the  sum  over  all  possible  discrete  values  of  x. 


The  limits  -»  to  »  apply  if  f(x)  is  defined  to  be  equal  to  zero  for  all 
impossible  values  of  x. 
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All  cumulative  distribution  functions  have  the  following  properties: 

F(X)  2  0  for  all  X 

F(x)  =  0  for  all  X  <  L.  the  lower  limit  of  the  range  of  x 
F(X)  =  1  for  all  X  §  U,  the  upper  limit  of  the  range  of  x 
F(X^)  §  F(X2)  if  X1  §  X2;  hence,  the  cumulative  distribution  function  is 


monotonically  increasing. 


4.4.2. 1  Example  A 


The  function  f(x),  shown  graph¬ 
ically  in  Sketch  A  and  stated  as 


f(x) 


i/2, 

1/4, 

1/8, 

1/8, 


x  =  1 
X  =  2 

x  =  3 
x  =  4 


Lb,  otherwise 


is  a  discrete  probability  density 
function,  since  f(x)  §  0  for  all  x  and 

/L  f(x)  =  1.0. 
x=l 


The  cumulative  function  is 


1.00 


x_ 


0.75 

0,50 

0.25 

0 


1 


r 

i 


i 

i 

i 


J _ I _ I _ L 

12  3  4 


5 


X 


Sketch  B 


F(X) 


ri/2,  x  = 
3/4,  x  =  2 
■7/8,  x  =  3 

L1*  x  =  4 


which,  when  plotted,  yields  the  step- 
function  shown  i  n  Sketch  B. 


The  probability  that  x  is  greater  than 
1  and'less  than  or  equal  to  3  is,  from 
Equation  19a, 


=  f (2 )  +  f(3)  =  1/4  +  1/8  =  3/8, 


or  is  equal  to  the  probability  that  x 
is  less  than  or  equal  to  3  minus  the 
probability  that  it  is  less  than  or 
equal  to  1  —  namely. 


F(3)  -  F(l)  =  7/8  -  1/2  =  3/8. 
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4. 4. 2. 2  Example  B 

The  function  f(x),  shown  graph¬ 
ically  in  Sketch  C  and  stated  as 

f2x,  0  s  x  s  l 

f(x)  = 

[_0,  otherwise 

is  a  continuous  probability  density 
function,  since  x  is  a  continuous  vari¬ 
able,  and  f(x)  2  0  for  all  x,  and 


=  1. 


The  cumulative  distribution  is 


p(X)  =  f 


f  (x)dx 


L 


2xdx  =  X 


X 

Sketch  D 


which  is  plotted  in  Sketch  D. 

The  probability  that  x  is  between  any 
two  values  in  the  range  of  x,  say 
between  a  and  b  is  (from  Equation  19b) 


rb 

/  f(x)d 
J  a 


X  =  F(b) 


P(a). 


Thus  the  probability  that  x  will  be 
between  0.3  and  0.6  is 

P  [0,3  <  x  s  0.6)  =  P(0.6)  -  F(0.3) 

=  (0.6)2  -  (0.3)2 


=  0.27 

4.4.3  Parameters  and  Moments 
4.4.3. 1  Definitions 

A  parameter  Is  a  constant  that  appears  in  the  probability  density  function. 
It  Is  more  generally  defined  as  some  measurable  characteristic  of  the  population, 
such  as  the  mean  or  range. 
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A  moment,  a  descriptive  property  of  a  probability  density  function,  is 
defined  as  follows: 


CD 


th 


=  « 


moment  about  zero: 

x  f(x),  for  discrete  variables 
L 


xrf (x)dx. 


for  continuous  variables 


(20a) 


(20b) 


moment  about  point  "a": 

V  (x-a)rf (x),  for  discrete  variables 

(21a) 

4_j 

X 

p  oo 

/  (x-a)  f(x)dx,  for  continuous  variables 

J  -OO 

(21b) 

—00 


The  first  moment  about  zero  (p.^  or  p.)  is  the  mean  of  the  distribution  and  is 
a  measure  of  central  tendency.  Mathematically,  the  mean  is  the  expected  or  aver¬ 
age  value  in  the  population;  it  is  defined  by  the  equation 


M-  = 


Vxf(x).  for  discrete  variables 
x 

/OO 

xf(x)dx,  for  continuous  variables 

-00 


(22a) 

(22b) 


The  second  moment  about  p  is  called  the  variance,  denoted  usually  by 
(p.gin  the  previous  notation).  It  is  a  measure  of  diapersion  about  the  mean. 
Mathematically,  the  variance  is  the  expected  or  average  value  of  the  square  of 
deviations  of  all  possible  values  from  the  mean;  it  is  defined  by 


2 

o  = 


(x-p)2f(x),  for  discrete  variables 


I  [  (x-p)2f  (x)dx,  for  continuous  variables 

c  - 


(23a) 

(23b) 


The  greater  the  variance,  the  mere  variability  there  is  in  the  distribution. 
The  square  root  of  the  variance  is  known  as  the  standard  deviation. 


4. 4. 3. 2  Relationship  of  Parameters  and  Moments  to  Reliability  Theory 

To  relate  the  above  concepts  to  an  important  area  of  reliability  --  namely, 
the  time- to-failure  density  function  and  the  reliability  function  --  let  t  denote 
the  random  variable  time- to- failure  and  f(t)the  time- to- failure  probability 
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density  function,  (f(t)  =  0  for  t  <  0).  The  reliability  over  a  time  interval  t, 
denoted  by  R(t),  is,  by  definition, 

R(t)  =  probability  that  failure  occurs  after  t  (the  reliability  function) 

=  1  minus  probability  that  failure  occurs  before  t 
=  1  minus  f  f (t')  dt'  (t'is  simply  a  dummy  variable  of  integration) 

- 

Since  the  derivative  of  the  cumulative  distribution  function  is  the  probability 
density  function  for  continuous  variates, 

'W  -^r1 ' 

then 

f(t)  =  Mil  =  ajigLtil  .  --d_R(fc) 

'  •  n  T”.  +•  nt- 


The  probability  that  an  item  will  fail  within  the  time  interval  to  tg  is 
equal  to  the  probability  that  it  will  fail  before  tg  minus  the  probability  that 
it  will  fail  before  t1;  or,  from  Equation  19b, 

P  [t1  <  t  s  tgj  =  f(t)dt 

=  P(t2)  -  F(t1) 

.[l-R(t2)]  -  [l-Wtp] 

=  R( )  "  R(t2)  (24) 

The  mean  time  to  failure  is,  from  Equation  22b, 

/CO 

}  tf(t)dt, 

which  for  most  density  functions  is  equivalent  to 

n  =  f”  R(t)dt 
o0 

The  variance  is,  from  Equation  23b, 

p  CO 

o2  =  /  (t-tx)2f(t)dt 
J  c 

C25) 
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For  reliability  problems,  the  following  definitions  are  important: 

Mean  Life.  The  first  moment  of  a  time-to-failure  density  function  —  i.e., 
the  average  (in  the  sense  of  arithmetic  mean)  time  that  an  item  will  function 
satisfactorily  before  failure. 

Mean  Time  to  Failure  (MTTF).  The  term  often  used  for  the  mean  life  of  non- 
repairable  items. 

Mean  Time  Between  Failures  (MTBF).  The  term  often  used  for  the  mean  life  of 
repairable  items. 

(Note:  The  reliability  for  a  time  period  equal  to  the  mean  life  varies  with  the 
type  of  failure  distribution;  e.g.,  the  reliability  at  the  mean  life  of  a  normal 
failure-time  distribution  is  0.5,  but  it  is  0.37  for  the  exponential  distribution.) 

Median  Life.  The  time  interval  for  which  there  is  a  0.50  reliability  (e.g., 
50  percent  of  items  that  have  been  life-te&ted  would  be  expected  to  fail  before 
the  median  life  and  the  other  50  percent  would  be  expected  to  fail  after  the 
median  life) . 

Failure  Rate.  The  rate  at  which  failures  occur  per  unit  time  in  the  interval 
t  to  t  +  h,  defined  by 


Mt;  h) 


(26) 


(Note:  The  term  "failure  rate"  can  be  confusing  because  it  is  used  in  various 
ways.  It  sometimes  represents  the  expected  proportion  of  failures  in  an  interval, 
provided  all  failures  are  instantly  replaced  —  especially  in  connection  with  the 
exponential  distribution,  which  is  discussed  in  a  subsequent  section.  Sometimes 
the  term  is  used  to  mean  the  conditional  probability  of  failure  during  an  interval, 
given  survival  at  the  beginning  of  the  interval,  in  which  case  the  divisor  h  in 
the  above  definition  is  omitted.  In  addition,  the  term  is  often  used  to  signify 
the  instantaneous  failure  rate  or  hazard  rate  as  defined  below,) 

Flazard  Rate.  The  instantaneous  failure  rate,  defined  as  follows: 

z(t)  =  lim  A  (t;  h) 
h-+  o 

-a  log  r( t ) 

dt 

=  ffl}  ^ 

Note  that  R(t)  can  be  shown  to  be  equal  to  the  following  expression: 


R(t)  =  e 


(28) 
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4.4.4  Discrete  Probability  Distributions 
4.4.4. 1  Hypergeometr lc  Distribution 

The  requirements  of  the  hypergeometric  distribution  are  as  follows: 

(1)  There  are  only  two  possible  outcomes  --  e.g.,  success  or  failure, 
defective  or  not  defective. 

(2)  There  is  a  finite  population  size. 

(3)  Sampling  is  performed  without  replacement  (dependent  trials). 

Assume  that  a  sample  of  n  items  is  drawn  from  a  population  of  size  N  that 
contains  1^5  successes  (an  integer)  and  N(l-p)  =  Nq  failures  (an  integer).  The 
hypergeometric  probability  density  function  gives  the  probability  of  obtaining 
x  failures  and  (n-x)  successes  in  the  sample.  It  is  expressed  as  follows: 


The  cumulative  distribution  is 


x  =  0,  1,  . . . ,  n 


(29) 


f  NpY Nq  \ 

p(x)  =  y 


k=o 


x  =  0,  1,  . . . ,  n 


The  mean  is  np,  and  the  variance  is  npq  • 


(30) 


If  N  is  large,  so  that  the  ratio  n/N  is  small  (say  <  .05),  hypcrgeometric 
probabilities  can  be  closely  approximated  by  the  binomial  probability  distribu¬ 
tion,  discussed  below. 


Example :  Lots  of  30  items  each  have  experienced  an 
average  portion  defective  of  20  percent,  or  6  defectives 
Thus,  Np  =  (30)  (0.20)  =  6,  Nq  =  (30)  (0.80)  =  24.  If  5 
items  are  sampled  from  a  log,  what  is  the  probability  of 
getting  (a)  exactly  2  defectives?,  (b)  2  or  fewer  defectives?, 
and  (c)  more  than  2  defectives? 


(a)  Exactly  two  defectives: 
From  Equation  29, 


f(2) 


6.5  24.23.22 
2,1  3.2.1 

30.29.28.27.26 


29 ,2b .27 . 

i.3.2'.l 


0.213 
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(b)  Two  or  fewer  defectives: 

P  [2  or  less  defectives]  =  F(2)j  from  Equation  30, 


=  0.298  +  0.447  +  0.213 
=  0.958 


(c)  More  than  two  defectives: 

P  [more  than  2  defectives]  =  1-P  [2  or  less  defectives] 

=  l-F(2)  =  1-0.958  =  0.042 

4. 4. 4. 2  Binomial  Distribution 

The  requirements  of  the  binomial  distribution  are  as  follows: 

(1)  There  are  only  two  possible  outcomes,  success  or  failure. 

(2)  The  probability  of  each  outcome  is  constant  for  all  trials. 

(3)  The  trials  are  independent  (equivalent  to  sampling  with  replacement). 

For  n  trials  with  constant  probability  p  for  success  and  (l-p)  for  failure, 
the  probability  density  function  for  obtaining  x  successes  is 

f(x)  =  (x)px  (l-p)n"x,  x  =  0,  1,  2,  ...,  n  (31) 

and  the  probability  of  x  or  fewer  failures  is  given  by  the  cumulative  distri¬ 
bution  function 
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Example:  Assume  that  past  experience  indicates  that 
parts  produced  from  a  continuous  production  process  yield 
5  percent  defective.  In  a  random  sample  of  30  parts  (30 
trials),  what  is  the  probability  that  2  or  fewer  defectives 
will  be  found? 


The  following  information  is  available: 

"Success"  =  nondefective  part 
Categories.  »Par.ure«  =  defective  part 

Probabilities:  Probability  of  success  =  q  =  1-p  =  0.95 
Probability  of  failure  =  p  =  0.05 


Sample  size,  n,  is  30 


Hence,  from  Equation  32, 

P  [k  *  2  ]  =  P  [k  =  0,  l,  or  2]  =  F(2) 


2 


=  o^y  (0.05)°  (0.95)30  +  y?§^T  (0.05)1  (0.95)29 
+  2^T  (°-°5)2  (0.95)28 


=  0.812 

4. 4. 4. 3  Poisson  Distribution 

The  Poisson  distribution  can  be  used  as  an  approximation  of  the  binomial 
distribution  or  as  the  distribution  of  number  of  independent  occurrences  in  a 
continuum,  such  as  time,  length,  or  volume. 

For  an  approximation  of  the  binomial,  the  conditions  are: 

Cl)  The  binomial  law  applies. 

(2)  The  sample  size,  n,  is  large;  and  the  probability  of  failure,  p,  is 
small.  A  practical  rule  of  thumb  is  p  £  0.10  and  np  z  10. 


The  probability  density  function  is 


f(x)  =  - — (x  6  o,  p>  o, 


The  parameter  np  represents  the  expected  or  average 
trials . 


n  >  o) 

number  of  failures 


(33) 

in  n 
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Example:  Assume  that  a  sample  of  25  items  is  selected 
from  a  large  lot  in  which  10  percent  of  the  items  are 
defective.  What  is  the  probability  of  two  defectives  in 
the  sample? 

From  the  Poisson  approximation  to  the  binomial. 


f  (2) 


e-(25)(0.10)  [f25„0.10),2 
2! 


=  0.2565 


The  binomial  probability  is 

f (2 )  =(  i)  (0.10)2  (0.90)23  =  0.2659 

If  the  Poisson  is  employed  as  the  distribution  of  the  number  of  independent 
occurrences  in  a  continuum,  such  as  time,  length,  or  volume,  the  conditions  are: 

(1)  The  number  of  expected  occurrences  (say  successes)  per  given  segment 
of  the  continuum  (e.g.,  an  interval  of  time)  I  s  a  constant. 

(2)  The  number  of  occurrences  produced  in  any  subsegment  is  independent 
of  the  number  of  occurrences  in  any  other  subsegment. 

(3)  No  meaning  can  be  ascribed  to  the  number  of  non-occurrences;  e.g., 
the  number  of  telephone  calls  not  made  during  a  day,  or  the  number 
of  non-dcfccts  in  a  sheet  of  steel,  cannot  be  evaluated. 

If  m  is  the  expected  number  of  occurrences  per  given  segment  of  the  continuum, 
the  probability  density  function  is 


f(x) 


X! 


(34) 


Both  the  mean  and  the  variance  are  equal  to  m; 

Example:  Assume  that  an  item  will  experience  an 
average  ot  a  failures  per  hour  if  each  failure  is  instantly 
repaired  or  replaced.  It  is  desired  to  find  the  probability 
that  x  failures  will  occur  if  this  item  is  life-tested  for 
t  hours  and  failures  are  repaired  or  replaced  with  identical 
items. 


If  A  is  the  average  number  of  failures  for  one  hour, 
then  m  =  At  is  the  average  number  of  failures  for  t  hours. 
Hence,  if  x  represents  the  number  of  failures  (occurrences), 
from  Equation  j4: 


f(x) 


x  =  0,  1,  2, 
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If  n  items  are  placed  on  test,  the  average  number  of  failures  is  m  =  nht. 


o  r 


f(x)  =  e~nM^n*t)x  ,  x  =  0,  1,  2,  ... 

If  h  =  0.001  per  hour,  t  =  50,  and  10  items  are  put  on  test,  then  m  =  nht  = 
10  (0.001)  (50)  =  0.5.  The  probability  of  observing  two  failures  is 


f(2) 


=  0.076 


4.4.5  Continuous  Distributions 

4.4.5. 1  Exponential  Distribution 

The  probability  density  of  the  exponential  distribution  is  given  by 

t  *  0,  e  >  0  (35a) 

x  =  l/g  (35b) 

where 


f(t)  < 


e~t/e 


=  \  e 


-7st 


Mean  =  9 

2 

Variance  =  9 

The  exponential  distribution  is  primarily  used  as  a  formula  for  waiting 
times,  or,  in  reliability,  as  a  formula  for  the  time-to-failure  density  function. 
The  latter  use  is  a  direct  consequence  of  assuming  that  the  probability  of  fail¬ 
ure  in  the  interval  t  to  t  +  h,  given  survival  to  t,  is  a  function  only  of  h,  the 
length  of  the  interval,  and  is  independent  of  the  age  of  the  product,  t.  This, 
in  turn,  implies  that  if  a  device  has  not  failed  after  some  period  of  operation, 
it  is  as  good  as  new,  which  is  equivalent  to  the  statement  that  the  hazard  rate 
of  the  exponential  is  a  constant  that  equals  the  reciprocal  of  the  mean  life, 
usually  denoted  by  A. 

The  reliability  function  is 

/OO 

f  (t)dt 
=  e_t/0 


(36) 
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At  the  mean  life,  8,  the  reliability  is 

R(t  =  0)  =  =  e"1  =  0.368 

Given  the  reliability  over  a  time  interval,  the  mean  life  can  be  found  from 
the  equation 


Figure  4-1  shows  the  exponential  reliability  function  with  time  given  in  0  units. 
A  short  table  of  the  exponential  is  given  in  Appendix  C. 


R  (t) 


0.5  1.0  1.5  2.0  2.5  3.0 

t  (e  units) 

FIGURE  4-1 

THE  EXPONENTIAL  RELIABILITY  FUNCTION 


The  exponential,  gamma,  and  Poisson  distributions  are  related.  For  the 
Poisson  distribution,  the  random  variable  is  the  number  of  failures  in  a  given 
time  interval.  For  the  gamma  distribution,  the  random  variable  is  the  time  to 
the  nth  failure.  For  the  exponential  distribution,  as  a  special  case  of  the 
gamma,  the  random  variable  is  the  time  to  the  first  failure.  "Poisson  process" 
is  a  term  used  to  encompass  these  situations. 
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The  mecnanism  underlying  the  exponential  reliability  function  is  tnat  the 
hazard  rate  (or  the  conditional  probability  of  failure  in  an  interval,  given 
survival  at  the  beginning  of  the  interval)  is  independent  of  the  accumulated 
life. 


The  use  of  this  type  of  "failure  law"  for  complex  systems  is  usually  justi¬ 
fied  because  of  the  many  forces  that  can  act  upon  the  system  and  produce  failure. 
Different  deterioration  mechanisms,  different  part  hazard-rate  functions,  and 
varying  environmental  conditions  often  result  in  stress- strength  combinations 
that  produce  failures  randomly  in  time  according  to  the  exponential  failure  law. 

Another  justification  for  the  exponential  in  long-life  complex  systems  is 
the  so-called  "approach  to  a  steady  state,"  wherein  the  hazard  rate  is  constant 
regardless  of  the  failure  pattern  'of  individual  parts.  This  state  occurs  as  a 
result  of  the  mixing  of  parts  of  different  ages  when  failed  elements  in  the  sys¬ 
tem  are  replaced  or  repaired.  As  an  example,  assume  that  a  system  contains  parts 
which  have  increasing  hazard  rates.  What  all  parts  are  new,  the  system  hazard 
rate  is  low;  it  increases  as  the  parts  age.  What  a  failed  part  is  replaced  by 
anew  one,  the  system  hazard  rate  decreases,  and  it  falls  sharply  when  a  number 
of  replacements  occur.  However,  it  will  again  start  to  rise  as  these  "second 
generation"  parts  begin  to  age.  Thus,  over  a  period  of  time,  the  system  hazard 
rate  decreases,  and  it  falls  sharply  when  a  number  of  replacements  occur.  How¬ 
ever.  it  will  again  start  to  rise  as  these  "second  generation"  parts  begin  to 
age.  Thus,  over  a  period  of  time,  the  system  hazard  rate  oscillates,  but  this 
cyclic  movement  diminishes  in  time  and  approaches  a  stable  state  with  a  constant 
hazard  rate. 

A  third  justification  for  assuming  the  exponential  distribution  is  that  the 
exponential  is  used  as  an  approximation  of  some  other  density  over  a  particular 
interval  of  time  for  which  the  true  hazard  rate  is  fairly  constant.  For  example, 
if  the  true  hazard-rate  function  is  as  shown  in  the  curve  below  (assume  that  the 
system  is  debugged),  assumption  of  an  exponential  for  the  period  from  0  to  250 
hours  will  give  a  reasonable  approximation. 


Time,  t 
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These  arguments  notwithstanding,  indiscriminate  use  of  the  exponential  (or 
indiscriminate  use  of  any  distribution,  for  that  matter)  can  lead  only  to  con¬ 
fusion  and  incorrectness.  It  is  therefore  obligatory  on  the  part  of  the  analyst 
to  validate  the  use  of  any  particular  distribution  function.  Broadly  speaking, 
there  are  two  approaches  to  validation:  ( 1 )  Historical  —  i.e.,  examination  of 
the  past  performance,  where  available,  of  the  item;  (2)  Statistical  —  i.e., 
"soundness  of  fit"  (Chi-square  and  Kolmogorov- Smirnov  tests). 

4. 4. 5. 2  Weibull  Distribution 

The  probability  density  of  the  two-parameter  Weibull  distribution  is  given 
by 

rt  R-l  -t^/“  ,  v 

f(t)  =  |  t3  1  e  (38) 


where 


Mean  =  a1//p  , 

rGr+1 


Variance  =  a(2/e+1) [F(2/B  +l)-r2(l/g+  1)] 


Characteristics 

The  flexibility  of  this  density  is  one  of  its  desirable  characteristics,  a  is 
a  scale  parameter  and  p  is  a  shape  parameter.  When  p  =  1.  this  distribution  reduces 
to  the  exponential. 

Reliability  Applications 

The  Weibull  distribution  is  receiving  wide  application  as  the  failure  pattern 
of  semiconductor  devices  and  mechanical  devices,  and  because  of  its  flexibility, 
it  is  also  being  used  to  describe  failure  patterns  at  the  unit  and  equipment 
leveis.  The  hazard  rate  is  constant  when  p  =  1,  is  an  increasing  function  of 
time  when  p  >  1,  and  is  a  decreasing  function  when  p  <  1. 

4 . 4 . 5 . 3  Gamma  Distribution 

The  probability  density*  of  the  gamma  distribution  is  given  by 

e-t/b  ,a-l 

f(t)  = - S-—  ,  t  §  0,  ago,  b  *  0  (39) 

(a- 1 )  !ba 


where 

Mean  =  ab 
Variance  =a/b2 

* 

r 00  -i 

For  "a"  not  an  integer,  (a-l)i  =  r(a)  =  /  e-y  ya_1  dy 

Jo 
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Characteristics 

The  critical  parameter  is  "a",  which  controls  the  shape  of  the  curve;  "b" 

is  a  scale  parameter  that  determines  the  abscissa  scale  (i.e.,  changing  b  merely 
narrows  or  broadens  the  curve).  When  a  is  equal  to  1,  the  distribution  reduces 
to  the  exponential. 

Rcliabilin  Applications 

The  puna  distribution  is  important  in  reliability  for  two  reasons.  First, 
it  is  an  extremely  flexible  distribution  and  can  therefore  be  used  to  fit  the 
failure  pattern  of  items  in  their  various  stages  of  development.  When  a  =  1,  the 
hazard  rate  is  constant.  It  increases  with  time  when  a  is  greater  than  one  and 
decreases  with  time  when  a  is  less  than  one. 

The  second  reason  is  that  the  estimated  mean  life  of  the  commonly  used  expo¬ 
nential  distribution  has  a  gamma  density,  which  can  be  used  to  make  probability 
statements  for  estimates  and  tests  of  the  mean  life. 

4. 4. 5. 4  Normal  Distribution 

The  probability  density  of  the  normal  distribution  is  given  by* 


f(t) 


<  t  <  oo 


(40) 


where 


Mean  =  d 
Variance  =  a2 


Characteristics 

The  normal  is  one  of  the  most  widely  used  continuous  densities.  The  density 
function  is  a  symmetrical  bell-shaped  curve,  as  shown  on  the  left.  The  cumulative 

function  is  not  directly  integrable, 
but  tables  of  the  normal  cumulative 
distribution  with  a  mean  of  zero  and 
a  variance  of  one  (called  the  standard 
or  normalized  form)  are  widely  avail¬ 
able.  These  tables  can  be  used  for 
any  normal  distribution  by  transferring 
the  original  t  variable  to  anew  vari¬ 
able,  y,  by  the  equation 

y  =  (41) 

^The  letter  t  will  be  generally  used  to  denote  the  random  variable,  which  is 
consistent  with  the  use  of  th'is  letter  for  failure  time. 
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The  variable  y  is  normally  distributed  with  a  mean  of  zero  and  a  variance  of  one. 
Thus,  to  find  the  probability  that  t  is  less  than  say,  c,  one  can  use  the  tables 

by  first  letting  y'  “  °  and  finding  the  probability  that  y  s  y  ,  from  the 

tables.  A  condensed  table  of  the  standard  normal  is  given  in  Appendix  C.  Under 
appropriate  conditions  the  normal  distribution  can  be  used  to  approxomate  the 
binomial  and  Poisson  probability  laws  (see  any  standard  statistical  text). 

Example:  Assume  a  =  100  and  a  =  5.  It  is  desired  to  find 
the  probability  of  obtaining  a  value  between  95  and  110  on  a 
single  trial. 


Let 

y,  =  =  _! 

y2  =  liO^lOO  =2 


Then 


P  [95  <  t  <  lioi  =  p  [-1  <  y  <  2] 
=  F (2 )  -  F(-l) 

=  0.977  -  0.159 
=  0.818 


Reliability  Applications 

Frequently,  the  normal  distribution  applies  to  items  in  which  the  failure 
occurs  as  a  result  of  some  wear-out  phenomenon,  since  the  hazard  rate  of  the  nor¬ 
mal  distribution  increases  with  time,  in  a  manner  consistent  with  a  wear-out  proc¬ 
ess.  Since  the  normal  distribution  implies  both  negative  and  positive  values,  it 
should  not  be  used  unless  one  of  the  following  three  conditions  is  met: 

(1)  a/a  *  3  (this  condition  establishes  that  the  probability  of  a  negative 
failure  time  is  small  enough  to  ignore) 

(2)  all  negative  times  observed  as  zero-hour  failures  are  the  result  of 
wear-out  during  production  testing,  checkouts,  installations,  etc. 
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(3)  A  truncated  normal  distribution  is  employed  that  distributes  the 
probability  area  from  to  0  over  the  positive  range  0  to  ».  The 
distribution's  density  is 


where 


f(t)  = 


0,  for  t  <  0 

bm  *-l/2  )2 *  *  > 


a  O  Jzft 


(42) 


1 

0  /27T 


e"1/2 


A  characteristic  of  the  normal  distribution  is  that  the  mean  life  and 
median  life  are  each  equal  to  the  time  interval  for  which  the  reliability  is  0.50. 


Example :  Assume  that  an  item  with  a  normal  tlme-to- 
f allure  distribution  has  a  mean  life  of  300  hours  and  a 
standard  deviation  of  40  hours  (p.  =  300  >  3cr  =  120).  Wlxt 
is  the  probability  that  this  item  will  operate  at  least 
250  hours  without  failing? 


R(250)  =  1-F(2S0) 


=  1 


1 

40  '/g?f 


If  y 


t-300 


the  limit  t  =  250  transforms  to 


250-300 

40 


-1.25 


and 


R (t=250 )  =  R(y=-1.25)  =  1  -  J  ‘1’25  -pL-  e"y2/2  dy 

-00 


=  1  -  0.106 

=  0.894 
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4. 4. 5. 5  Log-Normal  Distribution 
The  probability  density  of  the  log-normal  distribution  is  given  by 


where 


Ntan 

Median  =  ; 

Variance  =  e27 


+  40  2/2 

log  (median)  = 
+  a?  j-e4D2  -l) 


y 


(43) 


Characterist  ics 

If  the  logarithm  of  a  variable  is  normally  distributed,  the  variable  has  a 
log-normal  probability  distribution.  The  probability  density  function  is  posi¬ 
tively  skewed,  a  large  variance  being  associated  with  much  skewness.  Three  log¬ 


normal  distributions  with  identical 
means  but  different  variances  are 
plotted  at  the  left. 

Reliability  Applications 

The  hazard  rate  of  the  log-normal 
increases  with  time  until  the  mode 
(most  likely  failure  time)  is  reached 
(the  time  corresponding  to  the  maximum 
ordinate  of  the  density  function), 
after  which  it  decreases.  The  median 
life  is  the  more  convenient  and  usual 
measure  of  central  tendency  since  — 
unlike  the  me  a  —  it  is  independent 
of  the  variance. 


The  use  of  the  log-normal  distribution  has  been  found  to  reflect  adequately 
the  failure  pattern  of  many  semiconductor  devices  and  is  also  often  appropriate 
for  system  or  equipment  time-to-repair  distributions. 


4. 4. 5. 6  Other  Distributions 

There  are  many  other  Important  probability  distributions  that  have  not  been 
discussed.  Such  distributions  as  the  (chi  square)  and  the  t  are  often  used 
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for  inferential  purposes.  Some  of  the  specific  uses  of  these  distributions  in 
reliability  and  maintainability  analysis  are  discussed  in  the  following  section. 

4.5  STATISTICAL  ANALYSIS  OF  TEST  DATA  ■■  ESTIMATION 

This  section  reviews  important  statistical  aspects  of  the  evaluation  of 
tests  conducted  to  make  reliability  and  maintainability  inferences  about  a  popu¬ 
lation  of  items  through  estimation  procedures.  The  purpose  of  estimation  is  to 
describe  pertinent  characteristics  about  a  population  through  analysis  of  data 
on  samples.  The  two  major  approaches  to  estimation  are  as  follows: 

Nonparametric  estimates,  those  which  are  made  without  assumption  of  any 

particular  form  for  the  probability  distribution. 

Parametric  estimates,  those  which  are  based  on  a  known  or  assumed  distri¬ 
bution  of  the  population  characteristic  of  interest.  The  constants  in  the 

equation  that  describe  the  probability  distribution  are  called  parameters. 

As  an  example  of  these  two  approaches,  suppose  it  is  desired  to  estimate 
the  probability  that  an  item  will  survive  for  50  hours.  Twenty  sample  items  are 
tested  until  they  all  fail.  For  the  parametric  estimate  of  the  50-hour  survival 
probability,  R(50),  if  an  exponential  distribution  is  assumed,  R(50)  can  be 

A 

obtained  from  the  expression  e“5°/s >  where  6  is  the  estimate  of  the  mean  time  to 
failure  based  on  an  exponential  distribution  of  failure  times.  For  the  non¬ 
parametric  estimate,  the  estimate  of  R(50)  is  simply  the  proportion  of  the  sample 
that  survived  50  hours. 

Generally,  nonparametric  estimates  are  not  as  efficient  as  parametric  esti¬ 
mates,  since  the  former  require  greater  sample  sizes  to  achieve  the  same  precision 
as  the  latter.  Qr  the  other  hand,  since  no  assumption  about  the  population  dis¬ 
tribution  is  made  for  nonparametric  tests,  errors  arising  from  incorrect  assump¬ 
tions  are  not  encountered. 

The  three  common  types  of  estimates  are: 

(1)  Point  estimate  --  a  single-value  estimate  of  a  parameter  or  charac¬ 
teristic 

(2)  Interval  estimate  —  an  estimate  of  an  interval  that  is  believed  to 
contain  the  true  value  of  the  parameter  or  Characteristic 

(3)  Distribution  estimate  —  an  estimate  of  the  probability  distribution 
of  a  characterist i c 

The  most  common  type  of  point  estimate  is  the  maximum-likelihood  estimate, 
i.e.,  the  value  that  has  the  maximum  probability  of  producing  the  observed  sample 
results.  A  confidcncc-intcrval  estimate,  the  most  common  type  of  interval  esti¬ 
mate,  is  one  for  which  there  is  a  known  degree  of  confidence  (in  a  probability 
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sense)  that  the  true  value  of  the  unknown  parameter  or  characteristic  lies  within 
a  computed  interval  .  Whenever  possible,  a  confidence-interval  estimate  should  be 
given  along  with  the  point  estimate,  for  then  the  degree  of  precision  in  the  point 
estimate  can  be  assessed.  For  example,  assume  that  a  100-hour  MTHF  is  desired. 
Samples  of  two  different  designs  are  tested  and  the  results  are  as  follows: 

Characteristic  Item  A  Item  E 

A 

Point  Estimate,  6  95  hours  105  hours 

90$  Confidence  Interval  (  ©L>  6y)  (90-115)  (40-170) 

Although  the  point  estimate  for  Item  B  is  above  the  100-hour  requirement,  it 
is  seen  that  the  precision  of  the  estimate  as  determined  from  the  length  of  the 
confidence  interval  is  poor  in  comparison  with  Ijhat  of  Item  A.  In  this  case, 
since  it  is  more  certain  that  Item  A  will  be  close  to  or  exceed  the  requirement 
than  it  is  that  Item  B  will,  the  former  may  be  chosen.  If  only  the  point  esti¬ 
mates  were  considered,  the  reverse  decision  probably  would  be  made. 

Two  steps  are  generally  involved  in  making  a  distribution  estimate:  (1) 
hypothesizing  or  determining  through  data  analysis  the  form  of  the  distribution, 
and  (2)  making  point  estimates  of  appropriate  parameters  that  will  completely 
describe  the  distribution. 

Sections  4.5.1  and  4.5.2  summarize  various  types  of  estimation  procedures.  The 
general  approach  for  analyzing  test  data  for  estimation  purposes  consists  of  the 
following  steps: 

•  State  objectives  for  test  data  analysis 

•  Determine  appropriate  forms  of  statistical  estimates  to  meet  objectives 

•  Perform  any  necessary  preliminary  analyses  such  as  analysis  of  the  dis¬ 
tributional  form 

Determine  if  parametric  or  nonparametric  procedures  are  to  be  used 

•  Apply  appropriate  procedures  or  equations  to  obtain  estimates 

•  Note  unusual  data  results  and  set  up  test  plan  for  confirming  any  new 
hypotheses 

•  Report  on  results  completely,  describing  test  design,  data  collection, 
raw  data,  and  data  analysis 

4.5.1  Nonparametric  Estimation 

A  summary  of  various  nonparametric  estimates  is  presented  in  this  section. 
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4. 5. 1.1  Point  and  Interval  Estimates  of  Reliability  or  MaintainabiliU 

The  simplest  estimate  of  reliability  for  a  time  interval  (t), denoted  by 
R(t),  is  to  calculate  the  proportion  of  items  that  survive  over  that  time  inter¬ 
val.  Thus  if  n  items  are  put  on  test,  and  f  failures  occur  before  time  t, 

R(t)  =  (44) 

Similarly,  the  probability  of  completing  a  specified  maintenance  action  by  time 
t  is 


M(t)  =  £ 


(45) 


where  r  is  the  number  of  such  actions  completed  by  time  t  out  of  a  total  of  n 
repair  actions. 

These  equations  are  for  the  case  of  no  withdrawal  of  items  (censorship) 
during  the  test. 

Construction  of  a  confidence  interval  about  R(t)  or  M(t)  is  based  on  the 
fact  that  these  estimates  correspond  to  a  binomial  parameter.  The  equations  for 
confidence  limits  are  as  follows: 

Lower  (1  -  a)#  Limit* 


1 _ 

1  +  i  -:-4  F  (2f  +  2,  2n 
ft  “  f  a 


(46) 


where 


PQ  (2f  +2,  2n  -  2 f )  is  the  upper  a$  point  of  the  F  distribution 
with  2f  +  2,  and  2n  -  2f  degrees  of  freedom.  A  condensed  set  of 
F  values  is  presented  in  Appendix  C. 


Upper  (1  -  a  )#  Limit 


RU’  1-a 


1  + 


f  +  1 


Fa  (2n  -  2f  +  2,  ITT  2) 


(47) 


For  a  two-sided  (1  “  ct)$  limit,  the  Interval  is 


( 


L,a/2, 


V 


1  -  a/2 


(48) 


In  this  appendix,  (1-a)  %  is  to  be  interpreted  as  the  (1-a)  fractile  or, 
equivalently,  as  100  (1-a)%,  where  a  is  a  decimal. 
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Figure  C-l*  can  be  used  to  obtain  the  approximate  limits  directly  for  n  i  30. 
The  horizontal  axis  is  the  point-estimate  value,  e.g.,  R  or  M.  Starting  at  this 
estimate  and  proceeding  vertically  to  the  appropriate  sample-size  curve  and  then 
horizontally  to  the  Y  axis  gives  the  appropriate  confidence  limit. 


Example:  Assume  n  =  50  items  are  put  on  test  and  f  =  10 
failures  are  observed  before  60  hours.  Find  (1)  the  point 
estimate  of  R(6o),  (2)  the  lower  90^-confidence  limit,  and 
(3)  the  two-sided  90^-conf idence  limit. 


(1)  From  Equation  44, 


R(60)  =  5°' 10  =  o.80 


(2)  From  Equation  46, 


0.10' 


r  1/ 


l  + 


li 

TB5 


P0.10  <22>  8°) 


From  Table  C-4, 

P0.10  (22’8o>  -  1.5 

and 

r.  =  i/[i  +  n  (1-5  ) 

RL,0.10  L  2+oT  J 

~  0.71 

In  Figure  C-l,  for  R  =  0.8,  the  90^-lower-limit 
curve  yields  a  value  of  approximately  0.68. 

(3)  From  Equation  47, 


rl,0.05  -  1 


0.684 


Vo. 95 


l  / 


1 


P0.05  ^ 82, 20) 


0.888 


From  Figure  C-l,  the  approximate  90^  interval  is 
(0.69,  0.88) 


^Extracted  from  RADC  Reliability  Notebook. 
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The  equations  are  used  in  the  same  was  for  confidence  limits  if  maintenance¬ 
time  data  rather  than  failure- time  data  are  being  analyzed. 

4.5. 1.2  Reliability  Functions 

If  point  estimates  of  R(t)  are  made  for  various  values  of  t,  a  relationship 
of  R  to  time,  t,  which  is  the  reliability  function,  can  be  developed.  The  reli¬ 
ability  case  is  discussed  in  detail  here  since  essentially  the  same  procedures 
are  used  for  maintainability. 

N)  Censorship 

Two  methods  are  possible  when  no  items  are  censored  --  i.e.,  no  items  are 
withdrawn  for  reasons  other  than  failure. 

(1)  Estimation  at  fixed  points  in  time,  t^: 


R(t1) 


n-P(t±) 

- 


(49) 


where 


n  =  number  of  items  originally  on  test 
fj  =  number  of  failures  in  the  interval  tj_^  <  t  s  tj 

F(t1)  =  number  of  failures  occurring  on  or  before  ti 
(2)  Estimation  of  R(t)  at  failure  times.  tk: 

R(\)  =  -n-  '  *  t-1-  (50) 


where 


k  is  the  number  of  failures  occurring  on  or  before  the  ordered 
failure  time  tk;  i.e.,  the  failure  times  are  ordered  so  that 


Thus  if  the  fourth  failure  out  of  10  observations  occurs  at  20  hours, 
t4  =  20  and  R(  20 )  =  10  x~  \  T  1  =  7/n 

For  large  samples  (say,  n>  30),  Equations  49  and  50  yield  nearly  identical  results. 
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Exarple:  Ln  a  test  of  50  items,  failures  occurred  at  the 
following  elapsed  hours:  7,  18,  25,  27  ,  35,  4  1  47,  50,  54,60. 

Obtain  the  observed  reliability  function  at  (l)every  10-honr 
period  up  to  60  hours,  and  (2)' each  failure  time. 


Observed  reliability 
from  Equation  49: 

functions  at 

10-hour  intervals. 

tl 

fj  F(t±) 

n-F(t±) 

n-P(t1) 
R(ti>  -  „ 

0 

0  0 

50 

1.00 

10 

1  1 

49 

0.98 

20 

1  2 

48 

0.96 

30 

2  4 

46 

0.92 

40 

1  5 

45 

0.90 

50 

3  8 

42 

0.84 

60 

2  10 

40 

0.80 

Observed  reliability 
from  Equation  (50): 

functions  at 

each  failure  time. 

k 

n-k+1 

"(O  -  "iSf 

0 

0 

51 

1.00 

1 

7 

50 

0.980 

2 

18 

49 

0.961 

3 

25 

48 

0.941 

4 

27 

47 

0.922 

5 

35 

46 

0.902 

6 

41 

45 

0.882 

7 

47 

44 

0.863 

8 

50 

43 

0.843 

9 

54 

42 

0.824 

10 

60 

41 

0.804 

The  two  functions  are  plotted  in  Figure  4-2. 

Censorship 

If  terminated  or  censored  observations  occur,  and  censorship  takes  place  at 
fixed  times,  then 


R(t,) 


TT 

j=i 


n,  -  f. 

_J _ si 

f . 

3 


(51) 


4-33 


Reliabi lit 


AMCP  706-191 


L.  L. 

where  n^  is  the  number  of  items  starting  the  j  interval;  f  is  the  number  of 
failures  in  the  interval  tj).  At  tj_p  the  end  of  the  (j-l)st  interval, 

some  items  will  be  removed;  hence  rij  =  -  fj^  ■  wj-i  ^wj-i  =  number  of 

censored  items  at  tj^). 


1 . 00  p>s= 


0.70 

0.60 


-  Equation  (49) 

___  Equation  (50) 


where  n^  is  the  number  of  items  start¬ 
ing  the  interval.  If  more  than  one 


OBSERVED  RELIABILITY  FUNCTIONS  For  terminated  or  censored  items 

occurring  randomly  within  a  time  inter¬ 
val  between  failures. 


R(tl) 


1 

IT 

j=i 


Hj  -  w.,/2 


Vs 


+  1 


(53) 


where 


Wj  is  the  number  of  withdrawals  during  the  jfch  time  interval 


Example:  For  the  data  of  the  example  given  above  in  "No 
CensorsTfipTr7  assume  that  one  good  Item  was  withdrawn  every  ten 
hours.  Calculate  the  reliability  function  at  ten-hour 
intervals . 


The  following  values  are  derived  from  Equation  51: 


j 

Interval 
( hours! 

1l£i 

i 

fi(ti) 

1 

0  £  10 

l 

1 

50 

0.980 

1 

io 

0.980 

2 

10  £  20 

1 

1 

48 

0.979 

2 

20 

0.959 

3 

20  £  30 

2 

1 

46 

0.956 

3 

30 

0.917 

4 

30  S  40 

1 

1 

43 

0.977 

4 

40 

0.896 

5 

40  £  50 

3 

1 

41 

0.927 

5 

50 

0.831 

6 

50  £  60 

2 

- 

37 

0.946 

6 

60 

0.786 
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Example:  In  addition  to  the  failure  times  given  above  in  the 
example  of  "nN)  Censorship",  withdrawals  were  made  at  the 
following  times:  2,  5,  20,  22,  25,  47,  and  56  hours.  Calculate 
the  reliability  function  at  the  failure  times. 

The  following  values  are  derived  from  Equation  53: 


1 

Interval 

(hours') 

ii 

i 

W2 

/  c. 

.fcF 

R(fci) 

1 

0  s 

7 

2 

50 

49 

O.980 

7 

0.980 

2 

7  * 

18 

i 

0 

47 

47 

0.979 

18 

0.959 

3 

18  a 

25 

i 

3 

46 

44.5 

0.978 

25 

0.938 

4 

25  * 

27 

i 

0 

42 

42 

0.977 

27 

0.916 

5 

27  S 

35 

i 

0 

41 

41 

0.976 

35 

0  .894 

6 

35  s 

41 

i 

0 

40 

40 

0.976 

41 

0.873 

7 

41  a 

47 

i 

1 

39 

38.5 

0.975 

47 

0  .851 

8 

47  a 

50 

i 

0 

37 

37 

0.974 

50 

0.829 

9 

50  a 

54 

i 

0 

36 

36 

0.973 

54 

0.806 

10 

54  a 

60 

i 

1 

35 

34.5 

0.972 

60 

0.784 

4. 5. 1.3 

Maintainability 

-Functions 

Where  maintainability  is  concerned,  censorship  rarely  presents  a  problem 
since  observation  can  usually  be  continued  until  all  maintenance  actions  are 
complete.  The  nonpar ame trie  estimates  of  the  probability  that  a  maintenance 
action  will  be  completed  by  time  t  are  exactly  the  complement  of  the  reliability 
formulas  if  f,  the  number  of  failures,  is  replaced  by  r,  the  number  of  completed 
repair  actions  . 

Thus,  for  estimating  M(t)  at  fixed  points  in  time  the  following  is 
obtained  for  the  no-censorship  case: 


fiup.i  l  rj 

J-l 


(54) 


where 


n  ■  number  of  maintenance  actions  observed 

Tj  =  number  of  completed  repairs  in  the  interval  <  t  s  fcj 


For  estimates  at  repair  times, 

m  05) 

when  k  is  the  number  of  repair  actions  completed  on  or  before  the  ordered  repair 
time 
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Example:  Assume  that  nine  repair  actions  are  observed 
as  foTIowsT'0,2,  0.3,  0.4,  0.4,  0.5,  0.6,  0.8,  1.2,  2.0 


Then,  from  Equation  55,  the  following  Is  obtained: 


l.Or 


t 

k 

M[tl 

0.2 

1 

0.10 

0.3 

2 

0.20 

0.4 

4 

0.40 

0.5 

5 

0.50 

0.6 

6 

0 . 60 

0.8 

7 

0.70 

1.2 

8 

0.80 

2.0 

9 

0.90 

t 


4.5. 1.4  Confidence  Limits  for  Reliability  and  Maintainability  Functions 


Estimating  reliability  or  maintainability  at  fixed  points  in  time  without 
censorship  corresponds  to  estimating  a  binomial  parameter.  Equations  46,  47,  and 
48  can  be  used  to  obtain  limits  for  the  observed  functions  for  cases  of  no  censor¬ 
ship.  If  censorship  takes  place,  the  number  of  sample  items  varies.  If  the  total 
number  of  censored  items,  w,  is  small  compared  with  n  (say  w/n  <  0.10),  then,  aB  a 
rough  approximation,  the  sample- size  value  to  be  used  is  n-w/2. 


4. 5. 1.5  Measures  of  Central  Tendency  and  Dispersion 

The  usual  measure  of  central  tendency  is  the  mean  or  average  value;  and  the 
measure  of  dispersion  is  the  variance  or  its  square  root,  the  standard  deviation. 
For  the  nonparamctric  case,  these  measures  are  valid  only  if  the  data  are  not 
truncated  or  censored  --  that  Is,  for  the  reliability  case  all  sample  items  are 
tested  to  failure,  and  for  the  maintainability  case  all  started  repair  actions 
are  completed. 


If  represents  either  a  failure  time  or  a  repair  time,  the  mean  or  average 
value  is  estimated  by 


n 

7  -  H  J  (56) 

1=1 

where  n  is  the  number  of  observed  times.  The  variance  is  estimated  by 

n 

3  2  =  Tn^ry  £  ( V7-)2  (57) 

1=1 
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For  large  n  (say  greater  than  30)>  the  central- limit  theorem  may  be  used; 
this  states  that  the  quantity 


(58) 


has  approximately  a  normal  distribution  with  mean  0  and  variance  1  where  p  and  0 
are  the  population  mean  and  variance,  respectively. 

For  large  n,  s  can  be  used  as  an  estimate  for  a,  and  an  approximate  (1-  a)# 
two-sided  confidence  interval  for  p  is  obtained  from  the  equation 


where 


(T  - 


V 2,  * 


M-  s  T  +  t 


a/2, 


(59) 


t  n  is  the  a/2  percentage  point  of  the  t  statistic  with  n-1  degrees 

of  freedom.  These  values  are  tabulated  in  Table  C-2.  A  one-sided  limit 

is  obtained  by  replacing  ,  by  t  _  for  the  limit  desired. 

r  °  cl/*-,  n-1  J  a,n-i 

Example ;  Assume  that  30  failure  times  are  observed  and  that 
t  =  150,  s  =  4o.  Then  the  lower  95$  confidence  limit  for  the 
mean  failure  time,  9  ,  is 


S-t 


0.05,  29 


s//n 


s  a 


or 


®L,0. 05 


150-1.70 


40 

/3U 


137.4 


For  the  nonparametric  case,  the  more  usual  type  of  central-tendency  measure 
is  the  median;  for  dispersion,  it  is  the  difference  between  two  percentage  points 
on  the  estimated  distribution. 

Median-Point  and  Interval  Estimates 

The  median  is  that  value  which  divides  the  distribution  in  half.  Thus  the 
median  failure  time,  ^ ,  is  that  value  of  t  for  which 

R(t)  =  0.50 

The  estimate  of  tQ  is  obtained  by  constructing  the  reliability  or  maintaina¬ 
bility  function  by  the  methods  described  and  by  plotting  the  distribution  to  find 
the  value  of  t  for  which  R(t)  or  M(t)  =  0.50.  For  the  reliability  case,  this  pro¬ 
cedure  requires  that  testing  continue  until  at  least  half  of  the  items  fail. 
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Confidence  intervals  for  tQ  5(|  are  obtained  from  the  equation 


P 


K  < 


0.50 


<  t 


n-r+1 


n-r 


]-Z 


n! 

i ! ( n- 1 )  1 


(60) 


where  t  and  t  are  the  rfch  and  (n-n-l)th  observed  ordered  times  In  the 

sample.  Note  that  the  confidence  levels  that  can  be  used  are  restricted  to  the 
values  obtainable  from  the  right-hand  side  of  Equation  60. 

One-sided  limits  are  given  as  follows: 


i=0 


(61) 


(62) 


where 


ni  ..  n(n-l) (n-2) ■ * • (n-i-l) 
I ! (n-i ) !  ~  — L 


Example:  From  the  data  of  the  example  given  in  Subsection  4. 5.1. 3, 
the  median  repair  time  tQ  .50  0  •  5.  From  Equation  60, 


t3< 


*0.50 


7]  -  I  rr(f-i)!  (?) 


1=3 


0.8203 


Tables  of  the  binomial  distribution  can  be  used  to  evaluate  the  sum  on  the 
right  to  yield  a  90.82$  two-sided  interval  of  (0.4,  0.8),  For  a  one-sided  upper 
interval,  from  Equation  62, 


P 


^.50  < 


1  =  0 


9! 

tivftj 


=0.9805 

or  a  98.05$  upper  limit  for  t0  50  =  1.2. 
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P- Percent  Range:  A  Measure  of  Dispersion 

The  50#  or  interquatile  range  defined  by 

t0.50  =  *0.75  "  *0.25 

is  often  used  to  measure  dispersion  in  nonparametric  estimation  procedures.  For 
reliability  tp  is  the  value  of  t  for  which  R(t)  =  1-P,  while  for  maintainability 
it  is  the  value  of  t  for  which  M(t)  =  P.  tq  is  numt>er  °f  hours  over  which 
the  middle  50#  of  the  sample  observations  were  recorded.  For  the  data  of  Example  5, 
.  1.0  -  0.325  =  0.675. 

Values  of  P  other  than  50#  can  be  used.  For  example,  the  90-percent  range 
T  _  t  t  For  the  reliability  case,  with  truncated  (non-failed) items, 

tfts^-per^tSr?  Fan  £e°§an  be  used  if  only  the  minimum  value  of  R(t)  is  less  than 
0.50  and  the  maximum  value  of  P  is  [1-2  min  ft(t)]. 

4.5.2  Parametric  Estimates 

Statistical  estimation  procedures  baaed  on  a  known  or  assumed  form  of  the 
probability  distribution  function  are  presented  in  this  section.  The  character¬ 
istics  of  the  distributions  considered  here  are  discussed  in  Section  4.4, 

4.5.2. 1  Determining  the  Fonn  of  the  Distribution 

The  validity  of  parametric  estimates  depends  greatly  on  the  validity  of  the 
assumed  distributional  form.  In  some  cases,  the  knowledge  of  the  experiment  that 
produced  the  data  w  i  1 1  dictate  what  the  distribution  should  be.  For  example,  in 
testing  for  defects,  the  number  of  defective  items  in  a  sample  of  n  items  is  dis¬ 
tributed  binomially  if  each  sample  item  is  randomly  and  independently  selected 
from  a  lot  and  tested,  and  if  the  outcome  is  cither  good  or  defective.  In  most 
cases,  however,  there  is  no  indication  of  what  the  true  population  distribution 
is.  Two  fairly  simple  procedures  for  analyzing  test  data  to  determine  the  dis¬ 
tributional  fonn  are  presented  below.  These  procedures  are  called  goodness-of-fit 
tests . 


Graphical  Procedures 

The  graphical  procedures  for  goodness  of  fit  involve  plotting  the  sample 
distribution  and  comparing  it  visually  with  the  generic  forms  of  known  distribu¬ 
tion  functions.  To  aid  in  such  types  of  analysis,  special  graph  papers  have  been 
constructed  so  that  when  the  observed  distribution  is  plotted,  a  straight  line 
will  result  if  the  distribution  conforms. 

To  test  for  the  exponential  distribution,  where  R(t)  =  it  is  noted 

that  /n  R(t)  =  -t/e.  Thus  if  the  observed  reliability  data  conform  to  the  expo¬ 
nential  failure  law,  the  natural  logarithm  of  the  observed  reliability  function 
will  plot  as  a  straight  line  against  t. 
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Special  types  of  graph  paper  for  the  normal,  log-normal,  and  Weibull 
distributions  can  be  used  for  goodness-of-fit  tests. 

Kolmogorov-Smlr nov  Test 

The  Kolmogorov- Smirnov  test  is  an  analytical  procedure  for  testing  goodness 
of  fit,  although  the  easiest  means  for  performing  such  a  test  is  graphical.  The 
procedure  involves  comparing  the  observed  distribution  with  a  completely  specified 
theoretical  distribution  and  finding  the  maximum  deviation.  This  deviation  is 
then  compared  with  a  critical  value  that  is  dependent  on  a  preselected  level  of 
significance. 

The  steps  are  as  follows: 

(1)  Completely  specify  the  theoretical  distribution;  that  is,  if  the 
distribution  to  be  tested'  has  k  parameters  in  the  density  function, 
values  of  each  of  the  k  parameters  must  be  specified.* 

(2)  Obtain  the  observed  reliability  or  maintainability  function  and  plot 
on  a  graph. 

(3)  Find  the  critical  value  d  from  Table  C-5,  Appendix  C,  for  the  selected 
significance  level  and  observed  number  of  failures,  a  significance 
level  of  a  means  that  c$  of  the  time  the  test  will  reject  the  hypothesis 
that  the  distribution  conforms  to  the  one  under  test  when  in  fact  it 
does.  Often  this  is  stated  as  the  100  (l-a)$  confidence  level. 

(4)  Draw  curves  at  a  distance  of  d  above  and  below  the  specified  theoretical 
distribution.  These  curves  then  make  up  a  decision  band. 

(5)  Ch  the  same  graph,  plot  the  observed  distribution. 

(6)  If  the  observed  distribution  falls  completely  within  the  decision  band, 
the  conclusion  is  that  the  assumed  distribution  is  correct.  If  any 
point  of  the  observed  function  falls  outside  the  decision  band,  the 
assumed  distribution  is  rejected. 

In  many  cases,  one  is  interested  only  in  the  form  of  the  distribution  and 
has  no  basis  for  parameter  specification,  in  these  cases,  the  parameters  can 
be  estimated  from  the  test  data  to  obtain  the  theoretical  cumulative  function. 
However,  the  critical  d  values  in  Table  C-5  are  too  large  and  will  lead  to 
conservative  results  (lower  significance  level)  since  if  the  observed  d  value  is 
greater  than  the  critical  d  value,  there  is  high  assurance  that  the  hypothesized 
population  form  is  incorrect.  However,  the  chances  of  accepting  the  hypothesis 
if  it  is  false  is  also  increased.  Results  of  Monte  Carlo  investigations  have 
shown  that  the  following  adjustments  to  Table  C-5  can  be  made  to  yield  approx¬ 
imately  valued  critical  values  for  the  normal  and  exponential  distributions. 

Normal  distribution  -  estimate  u  &  o  from  the  data 
Multiply  d  values  in  Table  C-5  by  0.67 

*See  following  discussion  for  the  case  in  which  parameters  are  estimated  from 
the  data. 
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Exponential  Distribution  -  estimate  e  from  the  data 
Multiply  d  values  in  Table  C-5by  0.80 

Example ;  Assimie  that  the  following  failure  times  are 
observed  when  a  total  of  20  items  are  tested: 

18,  25  28,  39,  40,  48,  60,  66,  80,  81,  83,  96,  105, 

108,  130  (5  items  survived  past  130  hours) 

(1 )  Obtain  Theoretical  Distribution 

Suppose  a  test  is  being  made  for  an  exponential 

distribution.  Since  the  reliability  function  for  the 

-t/0 

exponential  is  R(t)  =  e"  ,  where  9  is  the  mean 
failure  time,  the  following  estimate  can  be  used: 

0  =  Total  Observed  Life* 

Number  of  Failures 

_  Total  Time  for  Failed  Items  +  Total  Thue  for  Non-Failed  Items 

Number  of  Failures 

e  m  9<2g_+^§52  =  107.5  hours 

Then  the  estimated  theoretical  reliability  function  is 
R(t) 

This  function  is  also  plotted  in  Figure  4-3. 

(2)  Calculate  Observed-Reliability  Function 

Equation  49  provides  the  following  calculation  for  the 
observed  reliability  function  (plotted  in  Figure  4-3): 


tk 

k 

n-k+1 

^ 

8 

1 

20 

0.952 

10 

2 

19 

0.905 

18 

3 

18 

0.857 

19 

4 

17 

0.810 

40 

5 

16 

0.762 

48 

6 

15 

0.714 

60 

7 

14 

0.667 

66 

8 

13 

0.619 

80 

9 

12 

0.571 

81 

10 

11 

0.523 

83 

11 

10 

0.476 

96 

12 

9 

0.428 

105 

13 

8 

0  .380 

108 

14 

7 

0.333 

130 

15 

6 

0.285 

*See  Subsection  4. 5. 2. 3. 
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FIGURE  4-3 

K0LM060R0V-SMIRN0V  TEST  FOR  EXPONENTIAL  DISTRIBUTION 


(3)  Obtain  Critical  d  Value 

From  Table  C-5j  Appendix  C.  the  unadjusted  critical  d 
value  for  a  sample  size  of  20  is  0,304  when  testing  is  being 
done  at  the  10$  significance  level.  By  use  of  the  correction 
factor  0.80  (since  Sis  estimated  from  the  data)  the  adjusted 
critical  value  becomes  0.243. 

(4)  Plot  Decision  Curve 

The  unadjusted  decision  curves  are  constructed  by  adding 
and  subtracting  0.304  to  the  theoretical  curve,  and  the 
adjusted  curves  are  obtained  by  adding  0.243  to  the  theoretical 
curve.  These  curves  are  also  shown  in  Figure  4-3* 
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(5 )  Decision 

Since  the  observed  reliability  function  falls  within 
the  decision  curves,  the  hypothesis  of  exponentiality 
cannot  be  rejected.  For  small  sample  sizes,  the  decision 
curves  are  quite  wide.  For  this  example,  it  is  likely  that 
other  distributions,  such  as  the  normal  or  Weibull,  will 
also  not  be  rejected. 

4, 5. 2. 2  Discrete  Distributions 

The  two  most  common  discrete  distributions  involved  in  reliability  and 
maintainability  testing  are  the  binomial  and  Poisson  distributions. 

Binomial 

The  random  variable,  x,  is  the  number  of  occurrences  of  an  attribute  in  n 
independent  trials  when  the  attribute  is  classified  by  either  of  two  mutually 
exclusive  categories.  For  reliability  and  maintainability,  the  attribute  of 
interest  is  normally  a  successful  outcome,  that  is,  non-failure  or  satisfactory 
repair  . 

The  probability  density  function  is  expressed  as  follows: 

P(x;  p,  n)  =(j)px(l-p)n'x  (64) 


where 


P(x;  p,  n)  =  probability  of  x  occurrences  in  n  trials  when 

the  constant  occurrence  probability  on  one  trial 
is  p. 

For  the  binomial  distribution,  the  mean  and  variance  are: 


Mean:  p  =  np 


where 


pis  the  expected  number  of  occurrences  in  n  trials 

o 

Variance:  a  =  np(l-p) 

Estimates  of  these  values  are  as  follows: 

2  =  £ 

*  n 

/v 

P  =  r 

o2  =  (n-r)  £ 


(65) 

(65A) 

(66) 


r  is  the  number  of  observed  occurrences  in  n  trials 


where 
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Equations  46,  47,  and  48  are  used  to  obtain  confidence  limits  on  p.  For 
n  i  30,  Figure  0-1,  Appendix  C.  can  be  used. 

Example:  In  a  test  of  50  items,  46  successes  were 

observed,  (That  is  the  point  estimate  of  success  proba¬ 
bility  and  the  associated  90^-eonfidence  limits? 

p  =  r/n  =  46/50  =  0.92 


From  Figure  0-1,  Appendix  C,  the  90#- confide  rice  interval  is 

(0.83,  0.97) 


Poisson  Distribution 

The  random  variable,  x,  is  the  number  of  occurrences  of  an  attribute  per 
unit  segment  (e.g.,  per  unit  time).  If  an  item  exhibits  a  constant  failure  rate, 
the  number  of  failures  in  a  fixed  period  of  time  is  Poisson- distributed  if  fail¬ 
ures  are  replaced  as  they  occur. 

The  probability  density  function  is  expressed  as  follows: 

-mt 

P(x;  m,  t)  =  e-  (67) 


where 


p(xj  m.  t)  =  probability  of  x  occurrences  in  t  segments  if 
Poisson  parameter  is  m 

m  =  the  mean  number  of  occurrences  per  unit  segment 

For  the  Poisson  distribution,  the  mean  and  variance  are: 

Mean:  mt 
Variance:  mt 


The  estimate  of  the  mean  is  ac  follows: 


where 


m 


r  is  the  number  of  observed,  occurrences  in  t  unit  segments 


(68) 


To  find  (l-  a.)  $  confidence  limits  on  m.  given  r  occurrences  in  t  unit 
segments,  mL  and  my  must  be  solved  for  the  following  equations: 
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r-1 

Z  e-"L»Lk  ,  <«) 

i,  ^  - n —  *  1  -  a 


Then 


For  an  upper  (1-  a)#  limit,  solve  for  My  in  the  equation 


z 


kT  ■ 


k=o 


a 


(70) 


Then 


“a 

“u  "  T~ 


For  two-sided  limits,  use  1“  a/2  and  a/2  in  Equations  69  and  70,  respec¬ 
tively.  Tables  of  the  Poisson  function  are  available  for  such  calculations.” 


Example:  Assume  that  ten  constanl-failurc-ratc  items 
are  put  on  test,  each  for  a  period  of  100  hours.  When  they 
fail,  they  are  replaced  by  new  items,  a  total  of  15  fail¬ 
ures  occurred.  Obtain  the  estimate  of  m,  the  mean  number 
of  failures  per  100-hour  interval,  and  obtain  the  95#- 
confidence  limits. 


The  estimate  of  m  is  r/t.  Since  15  failures  have  been 
observed  in  ten  100-hour  intervals. 


A 

m 


r 


n  = 
10 


1.5 


Hence,  1.5  failures  per  100  hours  can  be  expected. 

From  Equations  69  and  70,  the  95#-conf idence  interval  for  m  is 


(0.84  £  m  £  2,47) 


*R.  Ryswick  and  G.  Weiss,  Tables  of  the  Incomplete  Gamma  Function  of  Integral 
Order,  U.S.  Naval  Ordnance  Laboratory,  I960,  AST  1A  Number  AD2513Y7 . 
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4, 5, 2, 3  Continuous  Distributions 

Descriptions  of  and  estimation  procedures  for  continuous  distributions  are 
presented  in  this  section.  Specific  distributions  considered  are  the  exponential, 
normal,  log  normal,  and  Weibull. 

Exponential  Distribution 

The  random  variable,  t,  is  the  number  of  unit  segments  occurring  before  an 
event.  In  reliability,  t  represents  hours  or  cycles,  and  the  event  is  item  fail¬ 
ure.  In  maintainability,  t  can  be  maintenance  downtime,  and  the  event  is  the 
completion  of  a  maintenance  action.  It  is  assumed  here  that  t  represents  hours 
and  the  events  represent  failures. 

The  probability  distribution  function  is  expressed  as  follows: 


f{t;  A)  =  Ae'^ 


(71) 


where 


t  a  0  and  A  is  the  mean  number  of  failures  per  unit  time  (per  hour), 
commonly  called  the  failure  or  hazard  rate 

Since  A  is  equal  to  the  reciprocal  of  the  mean  number  of  hours  before  a  failure, 
the  following  can  be  written: 


f(t,  Q)  =  i  e't/e 


(72) 


where 


e  =  1/A  =  mean  failure  time 

For  the  exponential  distribution,  the  mean,  variance,  and  hazard  rats  are: 


e  =■  i/a 

(73) 

c2  =  1/A2 

(74) 

h(t)  =  a,  a  constant 

(75) 

Mean  failure  time,  6,  is  estimated  by  Procedure  I  or  Procedure  11. 
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Procedure  I:  Test  until  r  failures  occur: 

6  =  Total  Life  Observed  _  T 
Number  of  Observed  f  ailures  r 

To  obtain  T,  the  following  procedures  are  used: 

Procedure  la:  Replacement  Test  (failures  repaired  or  replaced) 

T  =  ntr 

where 

n  =  number  of  items  on  test 
t  =  time  at  which  the  rth  failure  occurred 
Procedure  lb:  Nonreplacement  Test 


r 

\ — i 


T  = 


t±  +  (n-r)tr 


(76) 


(77) 


(78) 


where 


ti  is  the  time  of  1th- ordered  failure 


Procedure  Ic;  Censored  Items  (withdrawal  or  loss  of  non-failed  items) 

c 


Failures  Replaced:  T  =  ^  tj  +  (n-c)tz 

J-l 


(79) 


where 


tj  =  time  of  j  -ordered  censorship 
c  =  number  of  censored  items 


Failures  Not  Replaced:  T  =  ^  +  L,  tJ  +  (n-r-c)tr 


(80) 


1=1 


.1=1 
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Procedure  11:  Testing  Terminated  by  Stopping  Rule  on  Test  Time. 

If  the  test  plan  is  such  that  the  test  terminates  after  a  specified 
number  of  test  hours,  t*,  have  accumulated,  it  is  possible  that  no  failures 
have  been  observed.  Then  Equation  76  cannot  be  used,  since  it  implies  that 
the  estimated  8  is  Infinite. 


In  general,  for  Procedure  II  testing,  if  the  number  of  failures,  r, 
is  small  (say  r  s  5)?  a  better  estimate  of  9  can  be  obtained  by  the  equation 


9  = 


(81) 


where 


T  is  calculated  as  in  Procedure  I  except  that  tp  is  now  replaced  by  t*, 
the  time  at  which  testing  is  stopped. 


Example  1:  Twenty  items  are  placed  on  test.  Testing 
continues  until  10  failures  are  observed.  Calculate  the 
estimated  mean  life  of  the  items  as  based  on  (l)a  replace¬ 
ment  test,  with  the  10th  failure  occurring  after  80  hours; 

(2)  a  nonreplacement  test,  with  failures  occurring  at  10,  ll, 
17 1  25,  31?  46,  52,  65i  79,  and  100  hours;  (3)  the  same  non- 
replacement  test  with  4  items  censored:  2  at  30  hours,  1  at 
50  hours,  and  1  at  60  hours. 

(1)  From  Equations  76  and  77, 


9  =  £  =  Sir  =  =  160  hours 

(2)  From  Equations  76  and  78, 

r 

£  t±  +  (n-r)t. 

9  =  I  =  i=1  = - 


=  =  144.2  hour 


(3)  From  Equations  76  and  80, 

£  fci  +  £  tJ  +  (n-r-c)tr 

»  _  lak _ isk _ 


_  442  +  170  +  6(100) 
10 


-  121.2  hours 
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Example  2;  Twenty  Items  are  placed  on  test  and  the  test  is 
terminated  after  100  hours.  Calculate  the  estimated  mean  life 
of  the  items  based  on  (l)a  replacement  test,  with  8  items 
failing  before  100  hours;  (2)  a  non- replacement  test,  with 
failures  occurring  as  in  Example  1(2), 

(1)  From  Equations  76  and  77, 

9  =  =  ^iioox  =  250  hours 

°  r  8 

(2)  Calculations  are  the  same  as  for  Example  1(2). 


Confidence- Interval  Estimates  on  e.  Two  situations  have  to  be  considered 
for  estimating  confidence  intervals:  one  in  which  the  test  is  run  until  a  pre¬ 
assigned  number  of  failures  (r*)  occur,  and  one  in  which  the  test  is  stopped  after 

a  preassigned  number  of  test  hours  (b*)  are  accumulated.  The  formula  for  the 

2 

confidence  interval  employs  the  x  (chi  square)  distribution.  A  short  table  of 
2 

X  values  is  given  in  Appendix  C.  The  general  notation  used  is 


X2(P>  d) 


where  p  and  d  are  two  constants  used  to  choose  the  correct  value  from  the  table. 


The  quantity  p  is  a  function  of  the  confidence  coefficient;  d,  known  as  the 
degrees  of  freedom,  is  a  function  of  the  number  of  failures.  Equations  82  and  83 
are  for  one-sided  and  two-sided  100  Cl"  a)  percent  confidence  intervals,  respec¬ 
tively.  For  nonrcplacemcnt  tests  with  a  fixed  truncation  time,  the  limits  are 
only  approximate.  These  confidence  limits  on  mean  life  are  as  follows: 


Confidence 

Interval 


Fixed  Number  of 
Failures,  r* 


Fixed  Truncation11 
_ Time  t* 


One-sided 
(Lower  L  imit ) 


2T 

2(a,2r) 


2T 


X  (a,2r  +  2) 


(82) 


Two-Sided  Limits 


/  2T  2T  \  f  2T 

Cx2(|,2r)  x2(l-f>2r ))  (x2(|>2r  +2)' 


2T _ 

X2d-  f,2r  +  2) 


) 


(83) 


b  For  non-replacement  tests,  only  one-sided  intervals  are  possible 
when  r  =  o.  Use  2n  degrees  of  freedom  for  the  lower  limit  if  all 
n  items  on  test  fail. 


Example  1 :  Twenty  items  undergo  a  replacement  test. 
Testing  continues  until  ten  failures  are  observed.  The  tenth 
failure  occurs  at  80  hours.  Determine  fl)the  mean  life  of 
the  items;  and  (2)  the  one-sided  and  two-sided  95 %  confidence 
intervals. 
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(l)  From  Equations  76  and  77, 


(2) 


0  =  ^  =  (20^°)  _  hours 

(so  that  T 

From  Equation  82, 

(  2T  \  (  2(1600)  \ 

(  3200  \ 

^x2(a,2r)'  }  Vx2(o.05,  20)'  ' 

1^31.41'  °°) 

=  (101.88,  «) 

(3)  From  Equation  83, 

/  2T  2T  \  ..  /  3200  3200  \ 

(,x2(f,2r)  '  x2(l-§,2r)J  (,34.17  '  9.591  ) 

=  (93.65,  333.65) 


Example  2:  Twenty  items  undergo  a  nonreplacem.ent 
test,  which  is  terminated  at  100  hours.  Failure  times 
observed  are  10,  16,  17,  25,  31,  46,  and  65  hours. 
Calculate  f  l)  the  one-sided  approximate  90$  confidence- 
interval  (a  =  0.10),  and  (2)  the  two-sided  approximate 
90$-conf idcncc  limits: 

(19  From  Equation  82, 


( 


2T 


X  (a,2r  +  2) 


,00^) 


(20-7) (100) 


1=1  _ 

X  2 { -10,  16) 


=  ( 128.29,“  ) 


(2)  From  Equation  83, 


2T _  2T  \  /  3020  3020  \ 

x2(|,  2r  +  2)  x2(l-|,2r)/  V.26.30'  6.57  2 

=  (114.83,  459.67) 
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Table  C-7,  Appendix  C.  extracted  from  the  RADC  Reliability 

2 

Notebook,  presents  the  factor  2/y  (p,  d)  for  one-sided  and  two- 
sided  confidence  limits,  at  six  confidence  levels  of  each. 
Multiplying  the  appropriate  factor  by  the  observed  total  life  T 
gives  a  confidence  limit  about  §. 


Sample-Size  Consideration.  Since  the  length  of  the  confidence  interval 
depends  on  the  number  of  failures,  it  is  possible  to  calculate  the  required  number 
of  failures  to  ensure  —  with  a  specified  confidence  __  that  the  estimate  of  0  i  s 
within  a  specified  percentage  of  the  true  mean  time  to  failure.  If  a  normal 

O 

approximation  to  the  X  distribution  is  used  in  order  to  be  (l-  a)$  confident 


that  0  is  within  6%  of  the  true  mean. 


8,  that  is  P 


1  "  a,  the  required 


number  of  failures,  r*,  is 


where  Zq  is  the  standardized  normal  deviate  corresponding  to  the  a.%  point  of  the 
normal  distribution.  Za  is  tabulated  in  Table  C-l,  Appendix  C. 

Once  r*  is  determined,  the  approximate  total  test  time  required  can  be  esti¬ 
mated  by  the  equation  T*  =  r*0',  where  01  is  an  initial  estimate  of  0. 

Example :  How  many  failures  arc  required  to  give  90-pcrccnt 
confidence  that  the  estimate  d  is  within  20-percent  of  the  true 
value?  What  will  be  the  total  test  time  if  0  =  100  hours? 

From  Equation  84,  for  90-percent  confidence,  Zq  =  ZQ  lQ  = 

1.645  and 


r*  .  IhSISf.  ,  67 

(0.20; 

If  8 '  =  100  hours,  then 

T*  =  r*0'  =  67(100)  =  6,700  hours 

Table  C-8,  Appendix  C,  presents  values  of  r*  for  selected 
confidence  and  precision  levels. 

Reliability  Estimates.  R(t),  To  estimate  the  probability  of  survival 
for  a  time  t,  the  estimates  of  8  (Equation  2-76)  can  be  used  in  the  equation 


R(t)  =  e_t,/® 


(85) 
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This  estimate  is  biased  (pessimistically  if  R(t)  >  l/Q  ~  0.367);  especially  if 
r  is  small.  An  unbiased  estimate  is 

R(t)  =  (l-t/l)r_1,  r>  1,  t<  T  (86) 


where  r  is  the  number  of  observed  failures  in  T  total  test  hours 


Example:  If  10  failures  are  observed  in  1600  hours, 

calculate  tjje  reliability  estimate  for  a  30-hour  period 
(note  that  6  =  160  hours; 


From  Equation  85, 

r(3o)  =  e~30/l6°  =  0.829 


Prom  Equation  86, 

R(30)  “  30A60o)9  =  0.843 


Confidence  Limits  on  R(t).  The  confidence  limits  on  e  can  be  used  to 
obtain  confidence  limits  on  R(t)  by  the  equation 


P 


A  A 

e"t/0L,a/2  s  R(t)  s  e"t/eU,l-a/2 


1  -  a 


(87) 


For  a  onc-sidcd  lower  limit, 


P 


e~t/0L,a  s  R(t) 


1  -  a 


(88) 


Example ;  For  a  mean  life  of  160  hours,  (l)what 
is  the  probability  of  an  item  surviving  100  hours? 
(2)  What  are  the  two-sided  95$  confidence  limits  on 
this  probability? 

(1)  From  Equation  85, 


8(100)  -  e~100/e  .  e"10o/16°  .  0.535 


(2)  From  Equation  87,  the  two-sided  95$  confidence  limits  are 


f  100  \ 

(^g-93.65,  g -$33.65 j  =  (0.344,  0.741). 
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Percentile  Estimates.  To^estimate  the  time  period.  for  which  there 
is  a  reliability  of  R,  the  estimate  TR  is 


T 


H  -  6  in  R 


(89) 


The  confidence  limits  on  T.  define  a  tolerance  Interval,  since  these  limits 

LX 

permit  the  statement  that  there  is  100  (1-  a)  percent  confidence  that  R  percent 
or  more  of  the  items  in  the  population  will  survive  TR  or  more  time  units.  -piie 
100  (1-  a)  percent  confidence  limits  on  TR  arfe  given  below: 


Confidence 

Fixed  Number 

Fixed  Truncation 

1 11  ter-v-a  1 

of  Failures:  r* 

Time,  t* 

One-sided 

/2T  in.  1/R 

(  T  in  1/R  \ 

(Lower  Limit) 

\*2(ct,  2r)  / 

XX  (ct,2r  +2)  / 

Two-sided 

Limits 


In  1/R  2T  in  1/R 
V  x^(f>  2r)  Vd-f^r) 


)( 


in  1/R 

X2  (§>  2r  +  2) 


,  jg.ifl  i/aOj 

x2(l-§,2r)  ) 


(91) 


Example:  For  a  mean  life  of  160  hours,  what  is  the 
estimated  time  period  for  which  the  reliability  is  0,80? 

What  are  the  95-pcrcent  one-  and  two-sided  confidence 
limits  on  TR? 

A 

Since  &  =  160  hours.  Equation  89  yields  TQ  gQ 
=  0  in  ^  =  160  (0.22314)  =  35.70  hours 

The  95-percent  one-sided,  and  two-sided  confidence  limits  on  Tr;  from 
Equations  90  and  91 ,  arc 

^2(1600)^.22314)  >(#)  =  (22.73,  «) 


and 


(  msoo^222m ,  (20.9o, 

Normal  Distribution 

The  normal  distribution  is  one  of  the  most  widely  used  continuous  densities 
because  (l>it  approximates  the  distribution  of  many  random  variables  and  (2) 
the  sample  estimates  tend  to  be  normally  distributed  with  increasing  sample  size. 
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If  an  Item  has  a  normal  distribution  of  failure  times,  its  failure  characteristic 
is  consistent  with  a  wearout  process. 

The  normal  probability  distribution  function  is  expressed  as  follows: 


e'  i  (^) 


-00  s  t  s 


(92) 


Its  mean,  variance,  and  hazard  rate  are  expressed  as  follows: 

Mean  =  \i 
Variance  =  0  2 

Hazard  Rate  (increases  with  t)  = 


where 


h(t)  - 


TtoI 


1  +^rjt)]  t  s  V- 


_ g(t) 

/7fa[l  -  r(t)]  *  >  ^ 


:(t)  =  /2  e"1/2  (iai)  ,  r(t)  =  [l  - 


_  2(t-M-/ 
G  7ra2 


1/2 


(93) 


Hie  mean  is  estimated  as  shown  in  Cases  1  and  II, 
Case  I  -  All  tested  items  fail: 


n 


and 

n  n  2 

-  o  i=l  1=1 

a c  -  - 

n(n-l) 


(94) 


(95) 


where 


t,  is  time  to  failure  of  the  it!l  Item 
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Case  II  -  Truncated  Test  (r  of  n  items  fall)  There  are  two  methods  for 
estimating  the  mean  --  the  graphical  method  and  the  regression  method: 

Graphical  Method  (r/n  >  1/2) 

(1)  Calculate  the  nonparametric  reliability  function,  R(t),  using  the 
most  appropriate  of  the  equations  (49  to  53). 

A 

(2)  Plot  R(t)  on  normal  probability  paper  and  fit  a  straight  line  yield¬ 
ing  the  estimated  normal  reliability  function  R^(t)  . 

Then 

*  "  *0.50  (96) 


where 


tQ  =  value  of  t  for  which  R^t)  =  0.50 
a  =  *0.50  “  *0.84 


(97) 


where 


*0  84  =  value  °f  *  for  which  R^t)  =  0.84 


Regression  Method 

(1)  )bta  by  an  pp  1  «  t  where  is  the 

1  failure  time. 


(2) 


t^,  find  the  normal  deviate 


For  each  failure  time, 
to  R(t^),  using  Table  C-l.  ^(t^)  corresponds  to  1-P(t 
R(ti)  =  0.971,  Z*  =  1.9;  for  R(t±)  -  0.50,  ZL  -  0. 


corresponding 
^)>  Thus,  for 


Then 


„  bd-ce 
P  ‘  b*-re  and 

~  cb-rd 


where 

r  =  number  of  failures 


(98) 

(99) 


*-  I 
1=1 


d 


I  Vi 

l-i 


e  - 


r 


1-1. 
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Example:  Assume  that  failure  data  are  generated  a a  shown 

in  Table  4-1  - 


Number  of  Completed 
Observations, 

Number  of  Censored 
Observations  at 

*1»  wi 

1 

4 

1 

3 

1 

4 

1 

3 

1 

3 

1 

3 

1 

4 

1 

4 

1 

4 

1 

4 

1 

3 

Total 

2ri  =  11 

2  =  39 

graphical  Method 

The  graphical  method  is  shown  in  Figure  4-4: 

£  =  ^0.50  =  4°°° 

0  =  *0.50  “  *0.84  “  4000-2740  =  1260 


Regression  Method 

A  A 

The  calculations  for  p.  and  a  using  Equations  98  and  99  are  shown  in  Table  4-2. 
The  two-sided  100  (1-  a.)  percent  confidence  interval  on  q  is 

(ff  “  *a/2,  r-1  X7T'  ^  +  *a/2,  r-1  x  Tr~>)  (10°) 

where  tay2  r_i  is  the  a/2  percentage  point  of  tne  t  distribution  with  r-1  degrees 

of  freedom.  This  interval  is  only  approximate  for  truncated  tests.  Values  of  t 
are  given  in  Table  C-2,  Appendix  C.  For  t  >  30.  Table  C-l,  Appendix  C.  of  the 
standardized  normal  deviate,  z,  can  be  used. 
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TABLE  4-2 

COMPUTATIONAL  PROCEDURE  FOR  ESTIMATING  THE  MEAN  AND  STANDARD 

DEVIATION  OF  A  NORMAL  DISTRIBUTION,  BASED  ON  ORDERED 

OBSERVATIONS  INVOLVING  CENSORSHIP 

Hours  to 

Observed  Reliability 

Normal  Deviate 

Failure, 

Function,  R(t^) 

Corresponding  to  R(t, ), 

Zi 

1300 

0  .980 

-2,06 

1692 

0.958 

-1.73 

2243 

0.935 

-1.51 

2278 

0.910 

-1.34 

2832 

O.883 

-1.19 

2862 

0.853 

-1.05 

2931 

0.819 

-0.91 

3212 

0.778 

-0.77 

3256 

0.726 

-0.60 

3410 

0.653 

-0.39 

3651 

0.522 

-0.06 

St1  =  29,667 

2Z±  =  11.61 

c  =  2t^  =  29,667  b  =  sz^=  -11.61  r  =  11 

d  =  Zt±z±  = 

-27,062.8  c  =  sz2i  =  15.77 

M.  =  =  3972 

b  -re 

0  =  cS.~I^d  =  1208 

b  -re 

Example:  From  the  data  for  the  preceding  example,  calculate 
the  95-percent  confidence  interval  for  0  . 

For  r  =  11,  and  a  =  0.05,  1Q  -  2.23 

Then, from  Equation  100,  the  95-percent  Interval  is 


3972  -  2.23  ~l,  3972  +  2.23 

’'TT  /T TV 


(3161,  4783) 
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Wcibull  Distribution 

Many  random  variables  of  failures  and  repairs  can  be  described  by  the 
Weibnll  distribution,  which,  because  of  its  three  parameters,  is  quite  flexible. 

The  Weibull  probability  distribution  function  is  expressed  as  follows  : 


f(t) 


a 


t  -z  y,  a,p,  7  >  0 


(102) 


where 

a  =  scale  parameter 

p  =  shape  parameter 

7  =  location  parameter 

The  location  parameter  y  represents  trie  minimum  failure  or  repair  time. 

Often  it  is  set  equal  to  zero,  and  the  density  is  then 

f(t)  =  e‘tP/a  t  *  0,  a,  p  >  0  (103) 

If  p  =  1,  the  Weibull  reduces  to  the  exponential.  if  P  is  known,  analysis 
may  proceed  exactly  as  for  tne  exponential  except  that  all  times  t±  are  replaced 
by  the  values  t^. 

The  mean,  variance,  and  hazard  rate  for  the  Weibull  distribution  are  as 
follows  : 

Meati  =  p  =  7  +  a1^  r(l/p  +  1)  (104) 


where 


T (l/p  +  l)is  the  gamma  function,  which  for  integer  values  of  (l/p  +  l)is 
equal  to  (l/p)i 


Variance 


2 

a  = 


2/P 


r(2/p  +  1)  -  r2(i/p  +  1) 


(105) 


Hazard  Rate  =  h(t)  =  t^_1 

a 

Note  : 

h(t)  decreases  with  t  if  p  <  1 
h(t)  is  constant  if  p  =  1 
h(t)  increases  with  t  if  p  >  1 


(106) 
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Estimates  of  \  ,  p,  and  a,  Analytical  procedures  are  available  for 
estimating  the  a  ,  pj  and  y  parameters  of  the  Weibull  distribution  from  test  data*, 
but  they  involve  fairly  complex  interative  procedures. 

A  relatively  simple  graphical  procedure  is  usually  used  to  obtain  estimates 
from  Weibull  probability  paper.  A  sample  of  such  paper  is  shown  in  Figure  4-6. 

Two  sets  of  scales  are  used.  Tne  left  scale,  F(t),  and  bottom  scale,  t,  are  for 
plotting  the  raw  failure  or  repair  data.  The  right  scale,  Y  and  the  upper  scale, 
X,  are  called  trie  principal  ordinate  and  principal  abscissa  scales  and  are  used 
for  obtaining  the  a  and  p  estimates.  The  principal  abscissa  is  that  horizontal 
line  for  which  X  =  0  on  the  right  scale,  and  the  principal  ordinate  is  that  verti¬ 
cal  line  for  which  Y  =  0  on  the  upper  scale. 

The  procedure  is  described  below7  for  the  case  in  which  r  failures  are  observed 
out  of  a  sample  of  n. 

(1)  Compute  tne  failure  probability  function  by  the  equation 

P^i^E-TT  U07) 


where 

t^  is  tne  time  of  the  1th  failure 
n  is  the  number  of  items  originally  on  test 


(2)  Plot  F(t1)  versus  t  on  Weibull  probability  paper  and  fit  a  smooth 
curve  through  the  points 

(3)  Estimate  7.  If  F(t1)  plots  as  a  straight  line,  *  =  0.  If  F^)  plots 
as  a  curve,  a  constant  value,  k,  is  to  be  subtracted  from  t±  such  that 
the  plot  of  F(t1-k)  versus  (t^-k)  is  best  fitted  by  a  straight  line. 

The  initial  value  of  k  can  be  either  the  first  failure  time,  t^,  or 
the  t  intercept  of  the  curve.  Several  values  of  k  may  have  to  be  tried 
before  a  reasonably  linear  plot  of  points  is  obtained.  yjie  estimate 
of  7  is  then  the  value  of  k  that  produces  a  linear  fit. 

(4)  Estimate  p.  The  estimate  of  P  is  the  slope  0f  the  fitted  curve.  It 
can  be  obtained  directly  from  the  equation 


where  Y0  Is  the  intercept  with  the  principal  ordinate  (Yq  <  0)  and  XQ  is 
the  intercept  nvith  the  principal  abscissa. 

*For  example,  D  Lloyd  and  M.  Lipow,  Reji ability:  Management  Methods  and 
Mathematics  ,  Prentice  Hall,  19 62,  pp" — 1?  /-1B1. -  - 
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(5)  Estimate  a.  The  intercept  of  the  fitted  line  with  the  principal 

ordinate  is  equal  to  -In  a.  Hence,  if  YQ  is  the  intercept  (which  is 

negative). 


a  =  G 


(109) 


a  can  also  be  obtained  from  the  equation  £n  a  =  p  In  t*,  where  t*  is 
the  value  of  t  for  which  F{t)  =  0.628. 


Y 0  or  Xo,  or  both,  may  lie  outside  the  graph: 

Cl)  Yq  outside  the  graph 

p  can  still  be  estimated  by  picking  any  two  points  on  X,  say  X1  and 
Xg,  and  finding  the  corresponding  Y's,  e.g.,  and  Yg.  Then 


A 

P 


Y2'Y1 


(no) 


a  is  then  estimated  by  the  equation 


/s 

a 


epxo 


(in) 


(2)  X„  outside  the  graph 

Since  a  docs  not  depend  on  XQ  and  p  can  be  obtained  by  Equation  110, 
this  case  presents  no  difficulties. 

(3)  XQ  and  Yo  outside  the  plot 

Multiply  the  t  scale  by  an  appropriate  power  of  10,  e.g.,  101,  10"1, 

p 

10  ,  etc.  The  slope  is  independent  of  the  scale,  and  therefore  p  is 

estimated  as  before.  The  estimate  for  a  is  obtained  by  the  equation 

a  =  10J£  a'  (112) 


where  j  is  the  scale  factor  and  a'  is  the  graphical  estimate  of  a  when 
the  data  are  plotted  on  the  basis  of  the  t  x  10^  scale. 

Another  possibility  is  that  the  Weibull  plot  appears  as  two  intersecting 
lines.  For  this  case,  two  sets  of  a,  p,  and  7  estimates  are  made,  one  for  each 
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linear  portion.  The  estimated  density  is  then 


f(t) 


P1(t-71)P1-1  e_(t-7i)Pi/a1 


(113) 


i  =  1  f or  t  s  t* 
i  =  2  for  t  >  t* 

and  t*  is  the  time  at  which  the  two  lines  intersect. 


Example:  Table  4-3  presents  failure  data  (grouped)  for 
germanium  power  transistors.  Seventy-five  transistors  were 
put  on  test  for  7000  hours  and  44  failures  were  observed. 
Failures  were  noted  every  250  hours  for  the  first  1000  hours 
and  every  1000  hours  thereafter.  Since  the  sample  size  is 
large,  the  formula  for  F(t)  can  be  slightly  modified  by  using 
n  in  the  denominator  of  Equation  107  rather  than  n  +  1. 

Step  (l) 

F(t1)  is  calculated  as  shown  in  Table  4-3. 


TABLE  4-3 

GERMANIUM  POWER  TRANSISTORS:  ACCUMULATIVE 

PERCENT  FAILURE  VS.  SELECTED  TIME  INTERVALS 

Fal  lure -Age 
( hours) 

Failures 

Accumulative 
Failures/Sample 
Size  of  75 

Accumulative 
Percent  Failure 

250 

17 

17/75 

22.7 

500 

8 

25/75 

33.3 

750 

1 

26/75 

34.7 

lOOO 

1 

27/75 

36.0 

2000 

0 

27/75 

36.0 

3000 

5 

32/75 

42.7 

4000 

3 

35/75 

46.7 

5000 

4 

39/75 

52.0 

6000 

3 

42/75 

56.0 

7000 

2 

44/75 

58.7 

Step  ( 2  ) 

Figure  4-7  shows  the  plot  of  the  data  on  Weibull  probability  paper.  Tjie 
t  axis  is  multiplied  by  10^  to  accommodate  the  failure  times,  a  straight 
line  fits  the  data  well 
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Scale  for  Princiwl  Abscissa 


Failure-age  (hours) 
[Auxiliary  Abscissa] 


FIGURE  4-7 

GRAPHIC  PROCEDURE  FOR  OBTAINING  ESTIMATES  OF  PARAMETERS 
ASSOCIATED  WITH  THE  WEIBULL  DISTRIBUTION 


Step  (3) 

A 

Since  the  points  arc  fitted  by  a  straight  line,  7=0. 

Step  (41 

The  intercept  of  the  fitted  line  with  the  principal  abscissa,  X0,  is 
approximately  2.85;  and  the  principal  ordinate  intercept,  Y0,  is  approxi¬ 
mately  -0.85.  Thus,  from  Equation  108, 


A 

P 


0.30 


Step  (5) 

The  intercept  of  the  fitted  line  with  the  principal  ordinate  is  approximately 
-O.85.  Hence,  from  Equation  109 » 

Q'  =  e“(“0.85)  =  e0*85  =  2.34 


and  the  unsealed  estimate  is,  from  Equation  112. 

S  .  103(0.30)  (2>34)  _  l8>6 
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Estimate  of  the  Mean  (i2 ) .  The  mean  can  be  estimated  by  replacing  the 
estimates  for  a,  p,  and  7  in  Equation  104.  Then 


ix  =  7  +  a  1/£  r(l/p  +  1)  (114) 

A  short  table  of  r  (X)  for  1  s  X  s  2  is  presented  in  Table  C-9,  Appendix  c.  The 
relationship  T(X  +  1)=  xr(x)  can  be  used  for  X>  2. 

Example:  Prom  the  data  of  the  preceding  example, 
estimate  q. 

From  Equation  114 

*  -  i«.«1/0'30  <Ar  +  » 

=  16,900  r  (4.33) 


Since 

r (4.33)  =  (3.33)  (2.33)  (1.33)  T(1.33)  =9.216, 
then 


=  (16,900)  (9.216)=  157,000  hours 


FIGURE  4-8 

THEORETICAL  WEIBULL  RELIABILITY  FUNCTION,  J(t),  FOR 
/3=0.30  AND  a=2, 34  (KILO-SCALE) 


Estimate  of  the  Reliability 
Function.  From  the  estimates  of  a, 

B,  and  7 , 

A 

R(t)  =  e"(t-7)P/£  (115) 

For  maintainability,  the  probability 
that  a  repair  is  completed  before  t 
hours  i  s 

fi(t)  =  1  -  (116) 

Example  ;  The  reliability  function 
for  the  data  or  Table  4-3  is  shown  in 
Figure  4-8. 
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Point  and  Lower  Confidence  Limit  on  R(t)*.  The  point  estimate  and 
lower  confidence  limit  on  R(t)  for  unknown  a  and  p,  when  r  failures  out  of  n  are 
observed,  are  obtained  as  follows: 


( 1 )  Compute 


I 


i=l 


in  t^  -  in  t 


(117) 


where 


H-l  bi in 

i=i 


t  is  the  time  period  of  interest 
ti  is  the  ith  failure  time 

a  and  are  constants  given  by  Johns  and  Lieberman,*  Table  11. 


(118) 


(2)  The  estimated  reliability  is 


VO 


ft(t)  =e 


i/zb 


(119) 


FIGURE  4-9 

WEIBULL  HAZARD-RATE  FUNCTION,  Z(t),  FOR 
/3=0.30  AND  a=2.34  [KILO-SCALE) 


(3)  Table  I  of  Johns  and  Lieberman* 
gives  the  value  of  R(t)L  1_Q,  the  (l-a)$ 

lower  confidence  bound  on  R(t),  obtained 
by  entering  the  table  with  the  calculated 
(Za/fcb)  value. 

Estimate  of  Hazard-Rale  Function. 

From  Equation  107,  the  estimated  hazard- 
rate  function  i  s 

A 

h(t)  =  1  (120) 

a 

Example:  The  hazard- rate  function 
for  the  data  of  Table  4-3  is  shown  in 
Figure  4-9. 


*M,V.  Johns  and  G.J.  Lieberman,  "An  Exact  Asymptotically  Efficient  Confidence 
Bound  for  Reliability  in  the  Case  of  the  Weibull  Distribution",  Technometrics, 
Volume  8,  Number  1,  February  1966,  - 
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Log-normal  Distribution 

A  random  variable  whose  logarithm  is  normally  distributed  is  said  to  have 
a  log-normal  distribution.  This  distribution  frequently  describes  repair-time 
distributions. 

The  log-normal  probability  distribution  function  Is  expressed  as  follows: 

1  -,/p  (in  t-x>)2/w2  .  .  n  (■, 

f(tj  u,  co)  =  cot  e  1/2  '  t  >  0  (1 


The  mean,  median,  variance,  and  hazard  rate  of  the  log-normal  distribution 


are: 


Mean:  |x  =  6 


x>  +  oi  /2 


(122) 


Median:  m  =  6 


(123) 


The  median  is  often  used  as  a  central-tendency  measure  for  the  log-normal 
since  it  is  independent  of  co. 


Variance:  a2  =  e2v>  +  “  (e“  “  "0 


(124) 


Hazard  Rate:  The  hazard  rate  of  the  log-normal  increases  until  the  mode 
^gu-a)2^  -s  rcachcd,  and  then  it  decreases. 

Estimates.  The  simplest  procedure  for  estimating  the  reliability  or 

maintainability  function  for  r  data  points  out  of  a  sample  of  n  is  to  employ 
log-normal  probability  paper.  By  fitting  a  straight  line  through  the  nonparametric 

function  (Equations  49  through  53),  the  following  estimates  are  obtained: 


u  m  t0.50 
m  =  t0.50 


(125) 

(126) 


where 

tn  is  the  time  for  which  R(t)  or  M(t)  =  0,50 

0  .pu 


For  co. 


a)  =  In  t 


0.50 


-  in  t 


0.84(0.16) 


(127) 


where 

Tq  0^  is  the  time  for  which  R(t)  =  0.84  or  M(t)  =  0.16 
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*  2/2 

h  =meW  (12a) 

a2  =  m2e“  (e“  -l) 

Confidence  Limits  on  Median.  Confidence 
from  the  equation 

P[V  "  ta/2i  .r-1  VF  ~  V  S  5  +  ta/2, 

where  t  /P  ,  is  the  (a/2)$  point  of  the  t  statistic  with  (r-l)  degrees  of 

freedom  (Table  C-2,  Appendix  C) .  This  represents  a  confidence  interval  on  the 
logarithm  of  the  time  for  which  reliability  or  maintainability  is  0.50.  Then 
for  the  median,  m  =  tg  for  a  (1  -  a)%  confidence  interval 

(eSL,  a/2  eS„,  l-a/s)  (1; 


(129) 


limits  on  v  can  be  obtained 


co 

r-l  7T 


=  1  -  a 


(130) 


where 


«L  and  uy 


are  lower  and  upper  limits  on  u , 


respectively. 


Example :  Forty-six  maintenance-action  times  on  an 
airborne  communications  receiver  are  shown  in  Table  4-4, 
along  with  the  nonparametric  maintainability  function. 

This  function  is  plotted  on  log-normal  probability  paper 
in  Figure  4-10. 

It  is  seen  that  a  straight  line  fits  the  data  points  fairly  well.  The  value 
of  tg  c-q  =  1.95.)  and  the  value  of  ^g  =  0.56. 


From  Equations  125,  126,  and  127, 


■o  =  in  tQ  =  0.668 


t0.50  “  1-95 


cd  =  in  t 


0,50  ”  ^O.l6  —  0. 668  +  0,579  — 


1.247 
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Then,  from  Equations  128  and  129, 

G  =  1.95  e°'78  =  4.25 
oZ  =  (1-95)2  61*562  (el.562  _x)  =  68>o6 

From  Equation  130, 

vL.  .025  '  0668  "  195  =  -3°8 

vU,  .975  =  0  668  +  196  H?  =  1028 


and  the  95-percent  confidence  interval  on  m  is,  by  Equation  131, 

(1.35,  2.80) 


TABLE  4-4 

COMPUTATIONS  FOR  MAINTAINABILITY  FUNCTION 

Observed 

Data 

Non- 

parametric 

Function 

Observed 

Data 

Non- 

parametric 

Function 

tl 

ri 

M  (t±) 

tl 

ri 

A  . 

M  (tA) 

1 

0.021 

3.3 

2 

0.681 

1 

0.043 

4.0 

2 

0.723 

4 

0.128 

4.5 

1 

O.745 

2 

0.170 

4.7 

1 

0.766 

0.7 

3 

0.234 

5-0 

1 

O.787 

0.8 

2 

0.277 

5.4 

1 

0.808 

1.0 

4 

0.362 

5.5 

1 

0.830 

1.1 

1 

0.383 

7.0 

1 

0.851 

1.3 

1 

0.404 

7.5 

1 

0.872 

1.5 

4 

0.489 

8.8 

1 

0.894 

2.0 

2 

0.532 

9.0 

1 

0.915 

2.2 

1 

0.553 

10.3 

1 

0.936 

2.5 

1 

0.574 

22.0 

1 

0.957 

2.7 

1 

0.596 

24.5 

1 

0.979 

3.0 

2 

0.638 

— - - - 

- - 1 
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A  FORTIORI  ANALYSIS.  An  analysis  deliberately  made  to  favor 
an  alternative  system  when  comp  .red  to  a  judgmental 
"best"  system.  If  the  "bestu  sysjtem  receives  a  favorable 
comparison  under  the  weighted  analysis,  its  position  is 
strengthened. 

ABSCISSA:  The  horizontal  distance  fror  the  vertical  axis 

of  a  graph,  usually  designated  x. 

ACCEPTANCE  SAMPLING:  Inspection  of  samples  of  incoming  lots 
to  determine  acceptance  or  rejection  of  the  lot.  It  is 
characterized  by  the  sample  size  n  and  the  acceptance 
number  c,  or  by  the  average  outgoing  quality  limit. 

ACCRUAL  ACCOUNTING:  The  recording  and  reporting  of  expenses 
as  the  operating  transactions  occur.  This  method,  in 
contrast  to  obligations  and  disbursements,  provides  a 
realistic  measurement  of  resources  consumed  in  doing 
the  work. 

ACCUMULATOR:  The  register  and  associated  equipment  in  the 

arithmetic  unit  of  the  computer  in  which  arithmetical 
and  logical  operations  are  performed. 

ACTIVE  REPAIR  TIME:  The  portion  of  the  down  time  during 

which  one  or  more  technicians  are  working  on  the  system 
to  effect  a  repair.  This  time  includes  preparation 
time,  fault  location  time,  fault  correction  time,  and 
final  check  out  time  for  the  system. 

ADDRESS:  An  identification,  represented  by  a  name,  label 

or  number,  for  a  register  or  location  in  storage. 

Addresses  are  also  a  part  of  an  instruction  word  along 
with  commands,  tags,  and  other  symbols. 

ADMINISTRATIVE  TIME:  The  portion  of  the  down  time  not  included 
under  active  repair  time  and  logistic  time. 

ALGORITHM:  An  orderly,  step-by-step  procedure  for  performing 

a  mathematical  operation  in  a  finite  number  of  steps. 

The  1040  form  is  an  algorithm  for  computing  personal 
income  tax . 

ALLOCATION:  (1)  The  distribution  of  available  resources 

to  the  various  activities  which  must  be  performed  in 
such  a  way  that  total  effectiveness  will  be  optimized. 
Allocation  is  necessary  when  there  are  limitations  on 
either  the  amount  of  resources  available  or  on  the  way 
in  which  they  can  be  expended  such  that  each  separate 
activity  cannot  be  performed  in  the  most  effective  way 
conceivable.  (2)  An  authorization  by  a  designated 
official  of  a  department  making  funds  available  within 
a  prescribed  amount  to  an  operating  agency  for  the 
purpose  of  making  allotments. 


^Reproduced  by  permission  of  the  U.  S.  Army  Management  School. 
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ALLOTMENT:  An  authorization  granted  by  an  operating  agency 

to  another  office  to  incur  obligations  within  a  specified 
amount  pursuant  to  an  appropriation  or  other  statutory 
provision  and  subject  to  specific  procedural,  bookkeeping, 
and  reporting  requirements. 

ALTERNATIVES:  The  means  by  which  objectives  can  be  attained. 

They  need  not  be  obvious  substitutes  for  one  another  or 
perform  the  same  specific  function.  Thus,  to  protect 
civilians  against  air  attack,  shelters,  "shooting" 
defenses,  and  retaliatory  striking  power  are  all 
alternatives . 

ANALOG  COMPUTER:  An  electronic  device  that  performs  mathe¬ 
matical  operations  on  numbers  which  are  expressed  as 
directly  measurable  quantities,  generally  voltages 
and  resistances.  Analog  computers  are  less  accurate 
than  digital  computers,  but  they  are  more  readily 
adaptable  to  changes  in  the  data  and  structure  of  a 
problem.  They  are  especially  well  suited  to  problems 
involving  differential  equations. 

ANALYSIS  OF  VARIANCE  (ANOVA)  :  The  basic  idea  of  ANOVA  is 
to  express  a  measure  of  the  total  variability  of  a 
set  of  data  as  a  sum  of  terms,  each  of  which  can  be 
attributed  to  a  specific  source  or  cause  of  variation. 

APPORTIONMENT:  A  distribution  made  by  the  Bureau  of  the 

Budget  of  amounts  available  for  obligation  or  expenditure 
in  an  appropriation  or  fund  account  into  amounts  avai labia 
for  specified  time  periods,  activities,  functions, 
projects,  objects,  or  combinations  thereof.  The  amounts 
so  apportioned  limit  the  obligations  to  be  incurred  or, 
when  so  specified,  expenditures  to  be  accrued. 

APPRAISAL:  Impartial  analysis  of  information  conducted 

at  each  responsible  management  and  control  level  to 
measure  the  effectiveness  and  efficiency  of  the  total 
process  and  determine  preventive/corrective  action. 

ARGUMENT:  (1)  An  independent  variable;  e,g,,  in  looking 

up  a  quantity  in  a  table,  the  number  or  any  of  the 
numbers  which  identifies  the  location  of  the  desired 
value;  or  in  a  mathematical  function,  the  variable 
which  when  a  certain  value  is  substituted  for  it  the 
value  of  the  function  is  determined.  (2)  An  operand 
in  an  operation  on  one  or  more  variables. 
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ARTIFICIAL  INTELLIGENCE:  The  study  of  computer  and  related 
techniques  to  supplement  the  intellectual  capabilities 
of  man.  As  man  has  invented  and  used  tools  to  increase 
his  physical  powers,  he  now  is  beginning  to  use  artificial 
intelligence  to  increase  his  mental  powers.  In  a  more 
restricted  sense,  the  study  of  techniques  for  more 
effective  use  of  digital  computers  by  improving  program¬ 
ming  techniques. 

ASSEMBLER:  A  computer  program  which  operates  on  symbolic 

input  data  to  produce  from  such  data  machine  instructions 
by  carrying  out  such  functions  as:  translation  of 
symbolic  operation  codes  into  computer  operating  instruc¬ 
tions;  assigning  locations  in  storage  for  successive 
instructions;  or  computation  of  absolute  addresses  from 
symbolic  addresses.  An  assembler  generally  translates 
input  symbolic  codes  into  machine  instructions  item  for 
item,  and  produces  as  output  the  same  number  of  instruc¬ 
tions  or  constants  which  were  defined  in  the  input 
symbolic  codes. 

AVAILABILITY'  :  The  probability  that  the  system  is  operating 
satisfactorily  at  any  point  in  time  when  used  under 
stated  conditions,  where  the  total  time  considered 
includes  operating  time,  active  repair  time,  administrative 
time,  and  logistic  time. 

AVERAGE  OUTGOING  QUALITY  LIMIT:  The  average  maximum  fraction 
defective  leaving  an  acceptance  sampling  plan. 

BALANCE  OF  INTERNATIONAL  PAYMENTS:  A  systematic  record  of 
the  economic  transactions  of  a  country  during  a  given 
period  which  involve  a  transfer  of-currency  between 
the  country's  residents  and  the  residents  of  the  rest 
of  the  world. 


BAYESIAN  STATISTICS:  Estimates  of  (prior)  probability 

distributions,  subsequently  revised  (posterior  dis¬ 
tribution)  to  incorporate  new  data  by  means  of  Bayes 
equation : 


p(a1|b)  = 


P(B|A1)  P(A-|_) 

p(b|ax)  p(ax)  +  p(b|a2)  p(a2) 


BERNOULLI  PROCESS:  A  random  process  that  yields  an  either-or 
outcome  on  each  trial  with  known  probability  of  occur¬ 
rence,  and  results  from  statistically  independent  trials. 
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BIAS:  An  unbalanced  range  of  error  such  that  the  average 

error  is  not  zero. 

BINARY:  A  characteristic,  property,  or  condition  in  which 

there  are  but  two  possible  alternatives;  e.g.,  the  binary 
number  system  using  2  as  its  base  and  using  only  the  digits 
zero  (0)  and  one  (1)  . 

BINOMIAL  DISTRIBUTION:  The  distribution  of  many  two-valued 
processes  such  as  heads  and  tails,  or  acceptable  and 
unacceptable  units. 

n  1  X  1”X 

Prob(x  heads  in  n  tosses)=  7  <  ; — r  P  ( 1  —  P ) 

vn-xj ■ x  a 


BIONICS:  The  application  of  knowledge  gained  from  the  analysis 

of  living  systems  to  the  creation  of  hardware  that  will 
perform  functions  in  a  manner  analogous  to  the  more 
sophisticated  functions  of  the  living  system. 

BIT:  A  unit  of  information  capacity  of  a  storage  device. 

BLACK  BOX:  An  unknown  and  often  unknowable  mechanism  or 

system  whose  operation  is  judged  solely  by  observation 
of  its  inputs  and  outputs. 

BOOLEAN  ALGEBRA:  A  process  of  reasoning,  or  a  deductive 

system  of  theorems  using  a  symbolic  logic,  and  dealing 
with  classes,  propositions,  or  on-off  circuit  elements. 

It  employs  symbols  to  represent  operators  such  as  AND, 

OR , NOT , EXCEPT ,  IF. ..THEN,  etc.,  to  permit  mathematical 
calculation. 

BRANCH:  The  selection  of  one  of  two  or  more  possible  paths 

in  the  flow  of  control  based  on  some  criterion.  The 
instructions  which  mechanize  this  concept  are  sometimes 
called  branch  instructions;  however,  the  terms  "transfer 
of  control"  and  "jump"  are  more  widely  used. 

BREAK-EVEN  POINT:  In  engineering-economic  studies,  the  point 
at  which  two  alternatives  become  equally  economical  by 
altering  the  value  of  one  of  the  variables  in  a  situation, 

BUDGET:  A  proposed  plan  by  an  organization  for  a  given  period 

of  time  reflecting  anticipated  resources  and  their  extimated 
expenditure  in  the  pursuit  of  objectives. 

BUILDING  BLOCK  COST:  One  kind  of  a  rough  estimate  of  the  cost 
of  an  alternative  for  planning  purposes.  The  estimate  is 
not  time-phased  and  does  not  provide  for  variations  such 
as  in  the  manning  of  the  unit  or  cost-quantity  relationships. 
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CENTRAL  LIMIT  THEOREM:  If  the  sample  size  is  large  (n£30), 
the  sampling  distribution  of  the  means,  X,  can  be 
approximated  closely  with  a  normal  distribution.  Further¬ 
more,  this  theorem  also  applies  when  n<30  provided  that 
the  distribution  of  the  population  from  which  the  samples 
are  taken  can  be  approximated  closely  with  a  normal  curve. 

CERTAINTY:  The  state  of  absolute  confidence  in  which  outcomes 

are  sure  and  predestined, 

CETERIBUS  PARIBUS:  The  assumption  that  all  conditions  other 

than  the  ones  specifically  being  analyzed  remain  constant 
or  unchanged. 

CHI  SQUARE  TEST:  A  statistical  test  for  relatedness  of  two 
discrete  variables,  say  height  and  weight  of  officers. 

CLEAR:  To  erase  the  contents  of  a  storage  device  by  replacing 

the  contents  with  blanks,  or  zeros. 

COMBINATIONS:  Number  of  possible  arrangements  of  n  elements 

taken  c  at  a  time  if  sequence  is  ignored. 


(n-c)lc! 


COMMENSURABILITY :  The  capability  of  two  qualities  or  values 

to  be  measured  by  a  meaningful  relevant  common  index. 

For  example,  machine  guns  and  rifles  are  commensurable 
either  in  dollar  cost  or  in  effectiveness,  e.g.,  enemy 
casualties.  However,  machine  guns  and  friendly  casualties 
are  not  conmensurable  in  terms  of  dollars, 

COMPILER:  A  computer  program  more  powerful  than  an  assembler. 

In  addition  to  its  translating  function  which  is  generally 
the  same  process  as  that  used  in  an  assembler,  it  is  able 
to  replace  certain  items  of  input  with  series  of  instruc¬ 
tions,  usually  called  subroutines.  Thus,  where  an  assembler 
translates  item  for  item,  and  produces  as  output  the  same 
number  of  instructions  or  constants  which  were  put  into 
it,  a  compiler  will  do  more  than  this.  The  program  which 
results  from  compiling  is  a  translated  and  expanded  version 
of  the  original . 

COMPUTER:  A  device  capable  of  accepting  information,  applying 

prescribed  processes  to  the  information,  and  supplying 
the  results  of  these  processes. 
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CONDITIONAL  PROBABILITY:  The  probabili ty-that  A  will  occur, 
given  that  B  has  occurred:  P (A (  B)  , 

CONFIDENCE:  The  degree  of  trust  or  assurance  placed  in  a 

given  result. 

CONFIDENCE  INTERVALS:  A  measure  of  effectiveness  in  testing, 
expressed  in  quantitative  terms;  e.g,,  the  value  of  a 
specific  factor  (variable)  lies  within  a  specified  interval 
95%  of  the  time. 

CONFIDENCE  LEVEL:  The  probability  that  the  true  value  of  a 
parameter  lies  within  a  stated  interval, 

CONSOLE:  A  portion  of  the  computer  which  may  be  used  to 

control  the  machine  manually,  correct  errors,  determine 
the  status  of  machine  circuits,  registers,  and  counters, 
determine  the  contents  of  storage,  and  manually  revise 
the  contents  of  storage. 

CONSTANT  DOLLARS:  A  statistical  series  is  said  to  be  expressed 
in  "constant  dollars"  when  the  effect  of  changes  in  the 
purchasing  power  of  the  dollar  has  been  removed.  Usually 
the  data  are  expressed  i  n  terms  of  some  selected  year: 
or  eet  of  years. 

CONSUMER  PRICE  INDEX.  A  measure  of  the  period-to  period 

fluctuations  in  the  prices  of  a  quantitatively  constant 

market  basket  of  goods  and  services  selected  as  represen¬ 
tative  of  a  specific  level  of  living.  Hence,  it  can  be 

thought  of  as  the  cost  of  maintaining  a  fixed  scale 
of  living. 

CONSTRAINT:  A  resource  limitation,  which  ma^  be  specific 

(e.g.,  the  supply  of  skilled  manpower  or  a  particular 
metal),  or  general  (e.g.  ,  total  available  funds), 

CONSUMER  RISK:  The  probability  of  accepting  an  item  which 
is,  in  fact,  unsatisfactory, 

CONTINGENCY  ANALYSIS:  Repetition  of  an  analysis  with  different 
qualitative  assumptions  such  as  theater,  or  type  of  con¬ 
flict,  to  determine  their  effects  on  the  results  of  the 
initial  analysis, 

CONTRACT  DEFINITION  PHASE  (CDP) :  The  specification,  in  com¬ 
peting  contracting  studies,  of  detailed  technical  per¬ 
formance  characteristics,  coats,  and  time-and-cost  sched¬ 
ules  for  engineering  development  and  production  of  a 
military  end  item. 
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CORRELATION:  In  a  general  sense  in  statistics,  correlation 

denotes  the  co-relation  or  covariation  between  two  variables. 

CORRELATION  COEFFICIENT:  A  number  that  attempts  to  measure 
the  interdependency  of  variables. 

COST.  Goods  or  services  used  or  consumed. 

COST  ANALYSIS:  The  systematic  examination  of  cost  (total 
resource  implications)  of  interrelated  activities  and 
equipment  to  determine  the  relative  costs  of  alternative 
systems,  organizations,  and  force  structures.  Cost  analysis 
is  not  designed  to  provide  the  precise  measurements  re¬ 
quired  for  budgetary  purposes . 

COST  CATEGORIES:  Three  major  program  cost  categories  are: 

(1)  Research  and  Development  .  Those  program  costs  pri¬ 
marily  associated  with  research  and  development  efforts, 
including  the  development  of  a  new  or  improved  capability 
to  the  point  of  operation.  These  costs  include  equipment 
costs  funded  under  the  RDT&E  appropriations  and  related 
Military  Construction  appropriation  costs.  They  exclude 
costs  that  appear  in  the  Military  Personnel f  Operation 
and  Maintenance,  and  Procurement  appropriations. 

(2)  Investment .  Those-program-costs  required  beyond 
the  development  phase  to  introduce  into  operational  use 
a  new  capability,  to  procure  initial,  additional,  or 
replacement  equipment  for  operational  forces  or  to  pro¬ 
vide  for  major  modifications  Of  an  existing  capability. 

They  include  Procurement  appropriation  costs  and  all 
Military  Construction  appropriation  costs  except  those 
associated  with  R&D.  They  exclude  RDT&E,  Military  Per¬ 
sonnel,  and  O&M  appropriation  costs. 

(3)  Operating.  Those  program  costs  necessary  to  operate 
and  maintain  the  capability.  These  costs  include  Military 
Personnel  and  O&M  appropriation  costs,  including  funds 
for  obtaining  replenishment  spares  from-stock  funds. 

They  exclude  RDT&E  and  Military  Construction  appropriation 
costs  . 

COST  EFFECTIVENESS  ANALYSIS:  The  quantitative  examination 
of  alternative  prospective  systems  for  the  purpose  of 
identifying  a  preferred  system  and  its  associated  equip¬ 
ment,  organizations,  etc.  The  examination  aims  at 
f'ndino  answers  to  a  question  and  not  at  justifying  a 
conclusion.  The  analytical  process  includes  trade-offs 
among  alternatives,  design  of  additional,  alternatives, 
and  the  measurement  of  the  effectiveness  and  cost  of  the 
altexnativee . 
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COST  ESTIMATE:  The  estimated  cost  of  a  component  or  aggregation 
of  components.  The  analysis  and  determination  of  cost  of 
interrelated  activities  and  equipment  is- cost  analysis. 

COST  ESTIMATING  RELATION  (CER) :  A  numerical  expression  of  the 

link  between  a  physical  characteristic,  resource  or  activity 
and  a  particular  cost  associated  with  it?  e.g.,  cost  of 
aircraft  maintenance  per  flying  hour. 

COST  INFORMATION  REPORTING  (CIR) :  A  uniform  system  for  col¬ 
lecting  and  processing  cost  and  related  data  on  major  items 
of  military  equipment,  Its  purpose  is  to  assist  both  industry 
and  government  in  planning  and  managing  weapon  systems 
development  and  production  activities. 

COST  MODEL:  An  ordered  arrangement  of  data  and  equations  that 
permits  translation  of  physical  resources  into  costs. 

COST  SENSITIVITY:  The  degree  to  which  costs  (e.g.,  total  systems 
costs)  change  in  response  to  varying  ' assumptions  regarding 
future  weapon  system  characteristics,  operational  concepts, 
logistic  concepts,  and  force  mix. 

CRITERION:  Test  of  preferredness  needed  to  tell  how  to  choose 

one  alternative  in  preference  to  another.  For  each  alter¬ 
native,  it  compares  the  extent  to  which  the  objectives 
are  attained  with  the  costs  or  resources  used. 

CYBERNETICS:  The  field  of  technology  involved  in  the  comparative 

study  of  the  control  and  intracommunication  of  information 
handling  machines  and  nervous  systems  of  animals  and  man 
in  order  to  understand  and  improve  communication. 

DECK:  A  collection  of  punched  cards,  commonly  a  complete  set 

of  cards  which  have  been  punched  for  a  definite  service 
or  purpose. 

DEPRECIATION:  Decline  in  the  value  of  capital  assets  over  time 

as  a  result  of  business  operation  and/or  technological 
innovation.  The  Internal  Revenue  Service  defines  depre¬ 
ciation  as  the  gradual  exhaustion  of  property  employed 
in  the  trade  or  business  of  a  taxpayer  —  sueh  exhaustion 
comprising  wear  and  tear,  decay  or  decline  from  natural 
causes,  and  various  forms  of  obsolescence-.  Accelerated 
depreciation  is  any  formula  for  depreciation  permitted 
by  the  IRS  that  provides  for  a  more  rapid  write-off  of 
reproducible  assets  than  would  be  possible  by  using  rates 
reflecting  true  economic  depreciatfon .  Accelerated  depre¬ 
ciation  provides  economic  incentives  for  investment  in 
plant  and  equipment. 
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DESIGN  ADEQUACY:  Probability  that  the  system  will  success¬ 
fully  accomplich  its  mission,  given  that  the  system  is 
operating  within  design  specifications. 

DETERMINISTIC  MODEL:  A  model  that  permits  no  uncertainty  in 
the  magnitudes  of  either  inputs  or  outputs.  An  example 
from  gunnery  is: 

W  =  RM  j  where 

1000 

W  is  the  lateral  distance  at  range  R:  R  is  the  range,  and 
M  is  the  angular  measure  in  mils  of  the  arc  subtended  by 
W  at  range  R.  For  any  set  of  given  values  for  R  and  M 
there  is  one  and  only  one  value  for  W.  Many  deterministic 
models  use  an  average  as  a  constant  value  input. 

DIGITAL  COMPUTER:  An  electronic  device  that  performs  mathe¬ 
matical  operations  on  numbers  which  are  expressed  as  digits 
in  some  sort  of  numerical  system. 

DIMINISHING  RETURNS:  An  increase  in  some  inputs  relative  to 
other  fixed  inputs  will  cause  total  output  to  increase; 
but  after  a  point  the  extra  output  resulting  from  the  same 
additions  of  extra  inputs  is  likely  to  become  less  and  less. 
This  falling  off  of  extra  returns  is  a  consequence  of  the 
fact  that  the  new  "doses"  of  the  varying  resources  have 
less  and  less  of  the  fixed  resources  to  work  with. 

DISBURSEMENTS:  The  amount  of  expenditure  checks  issued  and  cash 

payments  made,  net  of  refunds  received. 

DOCUMENTATION i  The  group  of  techniques  necessary  for  the 

orderly  presentation,  organization  and  communication  of 
recorded  specialized  knowledge,  in  order  to  maintain  a 
complete  record  of  reasons  for  changes  in  variables. 
Documentation  is  necessary  not  so  much  to  give  maximum 
utility  as  to  give  an  unquestionable  historical  reference 
record . 

DOWN  TIME:  Total  time  during  whfch  the  system  is  not  in  accept¬ 
able  operating  condition.  Down  time  can  in  turn  be  sub¬ 
divided  into  a  number  of  categories  such  as  active  repair 
time,  logistic  time,  and  administrative  time. 

DYNAMIC  PROGRAMMING:  In  a  multistage  decision  process,  a  sys¬ 
tematic  method  for  seatching  out  that  sequence  of  decisions 
(policy)  which  maximizes  or  minimizes  some  predefined 
objective  function.  The  method  is  based  on  Bellman's 
Principle  of  Optimality  which  states  that:  "An  optimal 
policy  has  the  property  that  whatever  the-initial  state 
and  initial  decision  are,  the  remaining  decisions  must 
constitute  an  optimal  policy  with  regard  to  the  state 
resulting  from  the  first  decision." 
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ECONOMETRICS:  The  branch  of  economics  that-nses  mathematics 

and  statistics  to  build  and  analyze  economic  models,  to 
explain  economic  phenomena,  and  to  estimate  values  for 
economic  variables.  The  statistical-methods  used  are 
especially  designed  to  deal  with  time^series  data. 

ECONOMIC  GROWTH:  The  sustained  increase  in  the  total  and  per 

capita  output  of  a  country  as  measured  by  its  gross  national 
product  (inconstant  prices)  or“other  output  statistics. 

ECONOMIC  LOT  SIZE:  The  cost-minimizing "size 'of ’order  to  buy 
or  batch  to  make. 

ECONOMIES  OF  SCALE:  Efficiencies,  usually  expressed  as  reduction 
in  cost  per  unit  of  output,  that  result-from  increasing 
the  size  of  the  productive  unit. 

ECONOMY:  Using  the  least  amount  of  resources  to  attain  a  given 

output  or  fixed  objective. 

EFFECTIVENESS:  The  degree  or  amount  of  capability  to  accomplish 

some  objective  (s)  ,  Various  criteria  (e-.g,-,  targets  de¬ 
stroyed,  tonnage  moved,  etc.)  might-be  used  to  provide 
a  measure  of  this  amount  of  capability. 

EFFICIENCY:  Attaining  the  greatest  possible  output  from  a  given 
amount  of  resources. 

EMPIRICAL  PROBABILITY:  The  observed  relative  frequency;  e,g., 
if  d  is  a  random  sample  of  size  n  drawn  from  a  stable 
universe  possessing  a  given  trait,  the  empirical  probability 
that  an  element  drawn  randomly  from  that  universe  is 
estimated  to  be  d/n, 

ESSENTIAL  ELEMENT  OF  ANALYSIS:  A  question  specifically  designed 
to  obtain  data  that  will  provide  an  answer  in  a  particular 
problem  area,  or  information  required  to  conduct  an  eval¬ 
uation  in  a  particular  functional  area. 

EXPECTED  VALUE:  The  probability  of  an  event  occuring  multiplied 
by  the  payoff  associated  with  its  occurrence. 

EXTERNAL  ECONOMIES:  Those  benefits  accruing  from  a  grouping  of 
industrial  activities  or  from  public  facilities,  One 
textile  plant  benefits  from  the  existence  of  several  textile 
plants  in  a  vicinity. 

EXTRAPOLATE:  Estimate  by  trend  projection  the  unknown  values 

that  lie  beyond  the  range  of  known  values  in  a  series. 
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FAILURE  RATE:  The  number  of  items  replaced  per  unit  time  due 
to  failure  of  that  item. 

FEASIBILITY  STUDY:  (1)  A  study  of  the  applicability  or  desir¬ 

ability  of  any  management  or  procedural  system  from  the 
standpoint  of  advantages  versus  disadvantages  in  any  given 
case;  (2)  also  a  study  to  determine  the  time  at  which  it 
would  be  practicable  or  desirable  to  install  such  a  system 
when  determined  to  be  advantageous;  (3)  a  study  to  deter¬ 
mine  whether  a  plan  is  capable  of  being  accomplished  suc¬ 
cessfully. 

FIELD  EXPERIMENT:  A  mode  of  research  involving  the  response  of 
personnel  in  a  field  situation  or  environment  to  a  test 
situation.  A  field  experiment  is  conducted  under  stat¬ 
istically  controlled  conditions  to  discover  the  capabilities 
and  limitations  of  some  military  plan,  organization,  or 
material  . 

FIXED  COSTS:  Those  elements  of  cost  that  do  not  vary  with 
volume  of  production. 

FIXED  POINT  ARITHMETIC:  (1)  A  method  of  calculation  in  which 

operations  take  place  in  an  invariant-manner,  and  in  which 
the  computer  does  not  consider  the  location  of  the  decimal 
point.  (2)  A  type  of  arithmetic  in  which  the  operands 
and  results  of  all  arithmetic  operations  must  be  properly 
scaled  so  as  to  have  a  magnitude  between  certain  fixed 
values  . 

FLOATING  POINT  ARITHMETIC:  A  method  of  calculation  which 

automatically  accounts  for  the  location  of  the  decimal 
point.  This  is  accomplished  by  handling  the  number  as  a 
signed  mantissa  times  the  radix  raised  to  an  integral 
exponent;  e.g.,  the  decimal  number  +88.3  might  be  written  as 

+0.88300000  x  102. 

FLOW  CHART3  A  graphic  representation  of  the  major  steps 

of  work  in  a  process.  The  illustrative  symbols  may  represent 
documents,  machines,  or  actions  taken  during  the  process. 

The  area  of  concentration  is  on  where  or  who  does  what 
rather  than  on  how  it  is  to  be  done. 

FORCE  STRUCTURE  ANALYSIS:  The  analysis  of  proposed  forces  to 
obtain  a  picture  of  resource  implications  for  planning. 

FORCE  STRUCTURE  COSTING:  The  determination  of  the  resource 

implications  (manpower,  materiel,  support,  training,  etc.) 
in  dollar  terms  of  a  given  force  structure  or  change  to  it. 
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FORCASTING:  Attempting  to  define  possible  courses  of  future 

events.  May  include  estimating  probabilities  associated 
with  each  course  of  events. 

FORTRAN:  A  programming  language  designed  for  problems  which 

can  be  expressed  in  algebraic  notation,  allowing  for 
exponentiation  and  up  to  three  subscripts .  The  FORTRAN 
■compiler  is  a  routine  for  a  given  machine  which  accepts  a 
program  written  in  FORTRAN  source  language  and  produces 
a  machine  language  routine  object  program.  FORTRAN  II 
added  considerably  to  the  power  of -the  original  language 
by  giving  it  the  ability  to  define  and  use  almost  unlimited 
hierarchies  of  subroutines,  all  sharing  a  common  storage 
region  if  desired.  Later  improvements  have  added  the  ability 
to  use  Boolean  expressions,  and  some  capabilities  for 
inserting  symbolic  machine  language  sequences  within  a 
source  program. 

FREE  TIME:  Time  during  which  operational  use  of  the  system  is 
not  required.  This  may  or  may  not  be  down  time,  depending 
on  whether  the  system  is  in  operable  condition. 

FREQUENCY  DISTRIBUTION:  An  arrangement  of  statistical  data  that 
divides  a  series  of  items  into  classes  and  indicates  the 
number  of  items  falling  into  each  class.  An  example  is 
the  income  distribution  in  which  the  number  of  persons 
falling  within  each  income  class  is  stated. 

FULL  EMPLOYMENT:  According  to  the  President's  Council  of 

Economic  Advisers,  the  full  employment  level  is  reached 
when  no  more  than  four  percent  of  the  civilian  labor  force 
is  unemployed. 

GAMING:  A  method  of  examining  policies  and  strategies  under 

the  conditions  of  a  particular  scenarfo,  allowing  factors 
(human  and  chance)  to  vary  in  the  scenario. 

GANTT  CHART:  A  chart  of  activity  plotted  against  time  usually 

used  to  schedule  or  reserve  resources  for  specific  activities. 

GROSS  NATIONAL  PRODUCT  (GNP)  s  Total  value  at  market  prices  of 
all  goods  and  services  produced  by  the  nation's  economy 
during  a  period  of  one  calendar  year.  As  calculated 
quarterly  by  the  Department  of  Commerce,  •ji.oss  national 
product  is  the  broadest  available  measure  of  the  rate  of 
economic  activity. 

GROSS  PRIVATE  DOMESTIC  INVESTMENT:  One  of  the  major  components 
of  GNP,  gross  private  domestic  investment  includes  annual 
outlays  for  producers'  durable  goods  (machinery  and  equipment9 , 
private  new  construction  of  both  residential  and  non-resid- 
ential  buildings  (including  those  acquired  by  owner/occupanta) , 
and  the  net  change  of  business  investment  in  inventories. 
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HEURISTIC:  Pertaining  to  systematic  trial  and  error  methods 

of  obtaining  solutions  to  problems. 

HISTOGRAM:  A  graphical  representation  of  a  frequency  distribution 

by  means  of  rectangles  whose  widths  represent  the  class 
intervals  and  whose  heights  represent  the  corresponding 
frequencies . 

HOLLERITH:  A  widely  used  system  of  encoding  alphanumeric 
information  onto  cards,  hence  "Hollerith"  cards  is 
synonymous  with  punch  cards. 

HOMOSTASIS:  tfhe  dynamic  condition  of  a  system  wherein  the  input 

and  output  abe  balanced  precisely,  thus  presenting  an 
appearance  of  no  change,  hence  a  steady  state. 

HUMAN  FACTORS  ANALYSIS:  Individual,  behavioral,  cultural,  or 
social  systems  and  their  relation  to  organizations,  pro¬ 
cedures,  and  material. 

HUMAN  FACTORS  ENGINEERING:  The  development  and  application  of 
scientific  methods  and  knowledge  about  human  capabilities 
and  limitations  to  the  selection,  design,  and  control  of 
operations,  environment,  and  material,  and  to  the  selection, 
and  training  of  personnel. 

HYSTERESIS:  The  lagging  in  the  response  of  a  unit  of  a  system 

behind  an  increase  or  a  decrease  in  the  strength  of  a  signal. 
It  is  a  phenomonon  demonstrated  by  materials  which  make 
their  behavior  a  function  of  the  history  of  the  environment 
to  which  they  have  been  subjected. 

IMPLIED  AND  INDUCED  OUTPUT:  Implied  output  is  that  which  can 
be  estimated  directly  from  the  nature  of  the  project  in¬ 
cluding  all  activities  without  which  the  project  could  not 
function.  Induced  output  covers  the  interindustry,  or 
intermediate,  requirements  of  those  activities  that  supply 
the  project  and  those  which  purchase  or  use  its  output; 
usually  measured  by  using  an  input-output  table. 

INCOMMENSURABILITY:  The  inability  of  two  qualities  or  values 
to  be  measured  by  a  meaningful  relevant  common  index. 

INCREMENTAL  COST:  The  added  costs  of  a  change  in  the  level  or 
nature  of  activity.  They  can  refer  to  any  kind  of  change: 
adding  a  new  product,  changing  distribution  channels,  adding 
new  machinery.  Although  they  are  sometimes  interpreted 
to  be  the  same  as  marginal  cost,  the  latter  has  a  much 
more  limited  meaning,  referring  to  cost  of  an  added  unit 
of  output . 
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INDIVIDUAL  SYSTEM  COSTING:  The  determination  of  the  total 
resource  implications  of  a  system  (organization)  with¬ 
out  consideration  of  the  interaction  of  the  system 
(organization)  as  part  of  a  force  structure. 

INDEX  NUMBER:  A  magnitude  expressed  as  a  percentage  of  the 
corresponding  magnitude  in  some  "base"  period.  The 
base  is  usually  designated  as  equal  to  100. 

INDIFFERENCE  MAP:  A  two-dimensional  graph  denoting  an  indi¬ 
vidual's  preference  system  with  respect  to  two  economic 
quantities.  The  body  of  the  graph  consists  of  a  family 
of  nonintersecting  lines  convex  to  the  origin.  Each 
line  of  the  family  represents  an  equally  desirable  mix¬ 
ture  of  the  quantities  in  question- 

INDUSTRIAL  DYNAMICS:  A  philosophy  relating  to  simulation  of  a  system 
conceived  as  a  network  of  flo/s  and  feedback  loops  interconnecting 
a  number  of  inventories  or  levels  and  responding  to  changes  in  its 
environment. 

INFLATION:  A  rise  in  the  general  level  of  prices. 

(Pure  inflation  is  defined  as  a  rise  in  the  general 
level  of  prices  unaccompanied  by  a  rise  in  output.) 

INFORMATION  SYSTEM:  A  combination  of  personnel,  efforts, 
forms,  formats,  instructions,  procedures,  data, 
communication  facilities  and  equipment  that  provides 
an  organized  and  interconnected  means--automated, 
manual,  or  a  combination  of  these--for  recording, 
collecting,  processing,  transmitting  and  displaying 
information  in  support  of  specific  functions. 

INFRASTRUCTURE  (SOCIAL  OVERHEAD  CAPITAL)  :  The  foundation 
underlying  a  nation's,  region's,  or  community's 
economy  (transportation  and  communications  systems, 
power  facilities,  schools,  hospitals,  etc.). 

I NPUT- OUTPUT  ANALYSIS :  A  quantitative  study  of  the  inter¬ 
dependence  of  a  group  of  activitiesbased  on  the  rela¬ 
tionship  between  inputs  and  outputs  of  the  activities. 
The  basic  tool  of  analysis  is  a  square  input-output 
table,  interaction  model,  for  a  given  period  that 
shows  simultaneously  for  each  activity  the  value  of 
inputs  and  outputs,  as  well  as  the  value  of  trans¬ 
actions  within  each  activity  itself.  It  has  been 
applied  to  the  economy  and  the  "industries"  into 
which  the  economy  can  be  divided. 

INSTRUCTION:  A  set  of  characters  which  defines  an  operation 
together  with  one  or  more  addresses,  or  no  address, 
and  which,  as  a  unit,  causes  the  computer  to  perform 
the  operation  on  the  indicated  quantities. 
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INTERACTION:  The  difference  between  a  whole  and-the  simple 

sum  of  its  parts. 

INTERCEPT:  Intersection  of  a  line  and  an  axis. 

INTERPOLATE:  Estimate  the  intermediate  value  in  a  series 

of  numbers  by  using  a  formula  that  relates  the  unknown 
value  to  the  pattern  of  known  valtcee  in  the  series, 

INTRINSIC  PROBABILITY:  The  probability  that  the  system  is 

operating  satisfactorily  at  any  point  in  time  when  used 
under  the  stated  conditions,  where  the  time  considered 
is  operating  time  and  active  repair  time. 

INVESTMENT  COST:  The  cost  beyond  the  Research  and  Develop¬ 
ment  phase  to  introduce  a  new  capability  into  operational 
use  . 

ISOCONTOURS:  Graphical  representation  showing  all  combinations 

of  inputs  that  produce  equal  outputs. 

ITERATIVE:  Describing  a  procedure  or  process--whi ch  repeatedly 

executes  a  series  of  operations  until  some  condition  is 
satisfied.  An  iterative  procedure  can  be  implemented 
by  a  loop  in  a  routine.  Each  iteration  or  cycle  used 
data  from  the  preceding  cycle  and  supplies  data  to  the 
following, 

ISOMORPHIC:  Similar  in  pattern. 

JOINT  COSTS:  Costs  that  are  shared  by  several  departments 

or  activities,  such  as  an  airbase  serving  fighter  squa¬ 
drons  and  transport  planes;  or  a  dam  providing  power, 
irrigation,  flood  control,  and  recreation. 

JOINT  PROBABILITY:  The  probability  that  both  event  A  and 

event  B  will  occur.  If  A  and  B  are  independent,  it  is 
the  product  of  their  separate  probabilities. 

KNOWN  UNIVERSE:  An  idealized  abstraction  from  the  real  world, 
in  which  the  probabilities  of  every  element  in  the  pop¬ 
ulation  are  known. 

LANGUAGE:  A  system  for  representing  and  communicating  infor¬ 

mation  which  is  intelligible  to  a  specific  machine. 

Such  a  language  may  include  instructions  which  define 
and  direct  machine  operations,  and  information  to  be 
recorded  by  or  acted  upon  by  these  machine  operations. 

LATIN  SQUARES:  Experimental  designs  to  avoid  compounding 
the  effects  of  inputs  while  reducing  the  number  of 
observations  (and  the  cost)  required  to  achieve  a 
satisfactory  confidence  level. 
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LEARNING  CURVE:  The  cost-quantf tp  relationships  far  estimating 
costs  of  equfpment.  Generally  used  to  predict  or  de¬ 
scribe  the  decrease  in  the  cost  of  a  unit  as  the  number 
of  units  produced  increases. 

LEAST-SQUARES  METHOD:  A  method  of  fitting  a  calculated  trend 
to  statistical  data,  so  called-because  the  sum  of  the 
squared  deviations  of  the  calculated  from  the  observed 
variables  is  a  minimum.  “Least  squares”  also  refers  to 
the  criterion  that,  when  followed,  yields  this  result. 

LIABILITIES:  The  amounts  owed  for  goods  and  services  received, 

other  assets  acquired,  and  performance  accepted.  This 
includes  amounts  administratively  approved 'for  payments 
of  grants,  pensions,  awards,  and  other  indebtedness  not 
involving  the  furnishing  of  goods  and  services. 

LINEAR  PROGRAMMING:  A  mathematical  method  used  to  determine 
the  most  effective  allocation  of  limited  resources  be¬ 
tween  competing  demands.  Mathematical  requirements  for 
applicability  of  linear  programming  are:  (1)  both  re¬ 

sources  and  activities  that  use  them  are  non-negative 
quantities;  and  (2)  both  the  objective  (e.g.,  profit  or 
cost)  and  the  restrictions  on  its  attaknment  are  expres¬ 
sible  as  a  system  of  linear  equalities  or  inequalities 
(y  =  a+bxj .  Linear  programming  has  been  employed  in 
areas  such  as  the  determination  of  the  best  product 
mix  and  the  selection  of  least-cost  transportation  routes. 

LOGARITHM:  The  logarithm  of  a  number  is  the  exponent  or 

power  to  which  the  logarithmic  base  must  be  raised  to 
equal  that  number. 

LOGARITHMIC  SCALE.  When  the  vertical  axis  of  a  chart  is 

laid  off  in  terms  of  the  logarithms  of  natural  numbers 
the  arrarigement  is  known  as  a  semilog  chart  and  the  ver¬ 
tical  scale  is  called  a  log  scale.  A  curve  plotted  on 
such  a  chart  represents  not  the  numbers  in  the  series 
but  the  logarithms  of  these  numbers.  Changes  in  the 
slope  of  such  a  curve  show  changes  in  the  percentage 
increase  or  decrease  of  the  original  series.  As  long 
as  there  is  no  change  in  direction,  equal  distances  on 
the  vertical  scale  correspond  to  the  same  percentage 
change  in  the  original  series. 

LOGISTIC  TIME:  That  portion  of  down  time-during  which  repair 
is  delayed  solely  because  of  the  necessity  of  mailing 
for  a  replacement  part  or  other  subdivision  of  the  system. 

LOOP:  A  self-contained  series  of  instructions  in  which  the 

last  instruction  can  modify  and  repeat  itself  until  a 
terminal  condition  is  reached. 
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MACHINE  LANGUAGE:  A  system  for  expressing  information  which 
is  intelligible  to  a  specific  machine.  Such  a  language 
may  include  instructions  which  define  and  direct  machine 
operations,  and  information  to  be  recorded  by  or  acted 
upon  by  these  machine  operations, 

MAINTAINABILITY:  Probability  that,  when  maintenance 

action  is  initiated  under  stated  conditions,  a  failed 
system  will  be  restored  to  operable  condition  within 
a  specified  total  down  time. 

MARGINAL  COST;  REVENUE:  Costs  incurred  or  expected  to 

be  incurred  in  the  production  of  an  additional  unit 
of  output.  Marginal  revenue  is  revenue  received 
or  expected  to  be  received  from  the  sale  of  an  ad¬ 
ditional  unit  of  output.  To  maximize  its  profits, 
a  firm  has  to  extend  production  to  the  point  where 
marginal  revenue  equals  marginal  cost. 

MARGINAL  OUTPUT  OR  PRODUCT:  The  output  to  be  derived 

from  the  use  of  an  additional  unit  of  a  productive 
resource  (land,  labor,  capital,  or  materials)  . 

MARGINAL  UTILITY:  Satisfaction  derived  from  the  last 
or  additional  expenditure.  Additional  increments 
of  expenditure  for  a  given  product  tend  to  result 
in  declining  additions  of  utility.  If  utility  is 
to  be  maximized,  the  satisfaction  derived  from  the 
last  dollar  spent  on  each  product  or  service  should 
be  the  same, 

MASTER  PLANNING  BUDGET:  The  estimated  cash  receipts  and 

disbursements  classified  as  to  causes  (contra  accounts) 
and  spread  over  the  future  periods  in  which  they  are 
predicted  to  occur.  For  comparability  with  other 
plans,  each  estimated  cash  flow  is  converted  into 
an  expected  value,  adjusted  for  risk  and  diminishing 
utility,  and  discounted  to  its  present  value. 

MATHEMATICAL.  MODEL:  The  general  characterization  of  a 

process,  object,  or  concept,  in  terms  of  mathematical 
symbols,  which  enables  the  relatively  simple  manip¬ 
ulation  of  variables  to  be  accomplished  in  order  to 
determine  how  the  process,  object,  or  concept  would 
behave  in  different  situations. 

MATRIX:  A  rectangular  array  of  terms  called  elements. 

It  is  used  to  facilitate  the  study  of  problems  in 
which  the  relation  between  these  elements  is  funda¬ 
mental.  A  matrix  is  usually  capable  of  being  subject 
to  a  mathematical  operation  by  means  of  an  operator 
or  another  matrix  according  to  prescribed  rules. 
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MEAN:  The  most  common  measure  of  central  tendency  equal  to 

the  sum  of  the  observed  quantities  divided  by  the  number 
of  observed  quantities  divided  by  the  number  of  observations. 

MEDIAN:  Halfway  point  between  the  two  end  points  of  an  array. 

MISSION:  The  specific  task  or  responsibility  that  a  person 

or  a  body  of  persons  is  assigned  to  do  or  fulfill. 

MISSION  RELIABILITY:  Probability  that  under  stated  conditions, 
the  system  will  operate  in  the  mode  for  which  it  was  d 
designed  for  the  duration  of  a  mission,  given  that  it 
was  operating  in  this  at  the  beginning  of-the  mission. 

MODE:  A  computer  system  of  data  representation.  The  value 

in  a  set  of  values  that  occurs  with  the  greatest  frequency. 

MODEL:  A  simplified  representation  of  an  operation,  containing 

only  those  aspects  of  primary  importance  to  the  problem 
under  study.  The  means  of  representation  may  vary  from 
a  set  of  mathematical  equations  or  a  computer  program 
to  a  purely  verbal  description  of  the  situation.  In 
cost/effectiveness  analysis  (or  any  analysis  of  choice), 
the  role  of  the  model  is  to  predict  the  costs  that  each 
alternative  would  incur  and  the  extent  wo  which  each  would 
attain  the  objective. 

MONTE  CARLO  method:  Any  procedure  that  involves  statistical 

sampling  techniques  from  a  distribution  of  possible  out¬ 
comes  for  obtaining  a  probabilistic  approximation  to  the 
solution  of  a  mathematical  or  physical  problem.  Monte 
Carlo  Methods  are  often  used  when  a  great  number  of 
variables  are  present,  with  inter-relationships  so  extremely 
complex  as  to  forestall  straightforward  analytical 
handling.  This  method  generally  involves  the  use  of 
simulated  data  acquired  by  putting  random  numbers  through 
transformations  such  that  the  data  imitates  significant 
aspects  of  a  situation. 

MONOTONICITY:  In  the  mathematical  sense,  monotonicity  refers 

to  the  constancy  of  a  type  of  change.  For  example,  i  f 
a  curve  is  rising  (falling)  throughout  the  range  of 
interest  we  say  it  is  a  monotonically  increasing  (decreasing) 
curve 

MOVING  AVERAGE:  A  series  of  averages  frequently  used  to 
reduce  irregularities  in  a  time  series  by  selecting 
a  set  number  of  successive  items  in  the  series, 
computing  the  average,  then  dropping  the  first  item 
and  adding  the  next  succeeding  one,  etc.  The  process 
is  intended  to  average  out  random  movements  and, 
thereby,  reveal  underlying  trends. 
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MUTUALLY  EXCLUSIVE:  Describing  any  event  the  occurrence  of 
which  precludes  the  occurrence  of  all  other  events 
under  consideration. 

NATIONAL  INCOME:  The  money  measure  of  the  overall  annual 

flow  of  goods  and  services  in  a  community  equal  to  the 
sum  of  compensation  of  employees,  profits  of  corporate 
and  unincorporated  enterprises,  net  interest,  and 
rental  income  of  persons.  is  also  equal  essentially 
to  GNP  minus  (1)  allowance  for  depreciation  and  other 
capital  consumption,  and  (2)  indirect  business  tax 
and  non-tax  liability  to  government. 

NUMERICAL  ANALYSIS:  The  study  of  methods  of  obtaining 

useful  quantitative  solutions  to  mathematical  problems, 
regardless  of  whether  an  analytic  solution  exists 
or  not,  and  the  study  of  the  errors  and  bounds  on 
errors  in  obtaining  such  solutions. 

OBJECTIVE:  The  purpose  to  be  achieved  or  the  position  to 

be  obtained.  Objectives  vary  with  the  level  of 
suboptimization  of  the  study. 

OBJECTIVE  FUNCTION:  A  mathematical  statement  of  goals, 
usually  profit  maximization. 

OPERAND:  A  quanity  entering  or  arising  in  an  instruction. 

An  operand  may  be  an  argument,  a  result,  a  parameter, 
or  an  indication  of  the  location  of  the  next 
instruction,  as  opposed  to  the  operation  code  or  symbol 
itself . 

OPERATING  COST:  The  recurring  cost  required  to  operate 
and  maintain  an  operational  capability. 

OPERATING  TIME:  Time  during  which  the  system  is  operating 
in  a  manner  acceptable  to  the  operator,  although 
unsatisfactory  operation  is  sometimes  the  result  of 
judgment  of  the  maintenance  man. 

OPERATIONAL  READINESS:  The  probability  that,  at  any  point 
in  time,  the  sytem  is  either  operating  satisfactorily 
or  ready  to  be  placed  in  operation  on  demand  when  used 
under  stated  conditions,  including  stated  allowable 
warning  time. 

OPERATIONS  RESEARCH:  The  use  of  analytic  methods  adopted 
from  mathematics  for  solving  operational  problems. 

The  objective  is  to  provide  management  with  a  more  logical 
basis  for  making  sound  predictions  and  decisions. 

Among  the  common  scientific  techniques  used  in 
operations  research  are  mathematical  programming, 
statistical  theory,  information  theory,  game  theory, 
monte  carlo  methods,  and  queueing  theory. 
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OPERATOR:  A  mathematical  symbol  which  represents  a 

mathematical  process  to  be  performed  on  an  associated 
operand . 

OPPORTUNITY  COST:  The  cost  of  foregone  opportunities;  the 
sacrificed  amount  of  money,  equipment,  or  units  of 
production  that  could  have  been  realized  by  a  separate 
course  of  action  (alternative)  with  the  same  time  and 
effort  expended. 

OPTIMIZATION:  The  attainment  of  the  best  possible  result, 

i„e.,  the  maximization  (minimization)  of  some  desirable 
(undesirable)  criterion  measure,  subject  to  the 
constraints  imposed  on  the  choice  of  solutions. 

ORDINATE:  The  vertical  distance  on  a  graph;  i,e. ,  the 

distance  from  the  horizontal  axis. 


PARAMETER:  A  constant  or  a  variable  in  mathematics  which 

remains  constant  during  some  calculation.  It  is  generally 
a  definable  characteristic  of  an  item,  device,  or  system. 

PARAMETRIC  ANALYSIS :  Parametric  analysis  assumes  a  range 
of  values  for  each  parameter  which  will  bracket  the 
expected  values  of  that  parameter,  and  a  solution  to 
the  problem  is  obtained  for  each  set  of  assumed  parameter 
values  . 


PAYOFF:  The ’gain  to  be  derived  if  a  particular  course  of 

events  develops  . 


PERIPHERAL  EQUIPMENT:  The  auxiliary  machines  which  may  be 

placed  under  the  control  of  the  central  computer.  Examples 
of  this  are  card  readers,  card  punches,  magnetic  tape 
feeds,  and  high-Speed  printers. 


PERMUTATIONS:  The  number  of  possible  sequences  of  n  items 

taken  c  at  a  time. 


P(n)  =  n!  P(")  -  m-c)  :  =  C(“)cl 


nl 


n 
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PLANNING:  The  selection  of  courses  of  action  through  a 

systematic  consideration  of  alternatives  in  order  to 
attain  organizational  objectives. 

PLOTTER:  A  visual  display  or  board  controlled  by  a 

computer  in  which  a  dependent  variable  is  graphed  by 
an  automatically  controlled  pen  or  pencil  as  a  function 
of  one  or  more  variables. 

PRESENT  VALUE:  The  estimated  present  worth  of  a  stream  of 
future  benefits  or  costs  arrived  at  by  discounting  the 
future  values,  using  an  appropriate  interest  rate. 

PROBABILISTIC  MODEL:  A  model  that  makes  allowances  for 

randomness  in  one  or  more  of  the  factors  that  determine 
the  outputs  of  the  model.  For  example,  an  inventory 
model  that  optimizes  an  inventory  policy  to  avoid  inventory 
shortages  is  probabilistic  if  it  takes  explicit  account 
of  uncertainty  over  time,  in  the  distribution  of  demands 
on  the  inventory.  On  the  other  hand,  the  model 
would  be  deterministic  if  it  assumed  that  the  rate  of 
demand  against  the  inventory  is  always  the  same  (usually 
the  estimated  average  demand) .  In  this  example,  a 
deterministic  model  would  most  probably  give  answers 
that  would  lead  to  bad  inventory  policies.  However, 
there  are  times  when  the  use  of  a  deterministic  model 
in  a  probabi lisitc  situation  does  no  harm. 

PROBABILITY:  A  number  between  0  and  1  that,  when  assigned 

to  an  event  or  occurence,  expresses  the  likelihood  that 
the  event  will  occur. 

PROBABILITY  DISTRIBUTION:  Tables  showing  relative 

frequencies  of  each  subset  into  which  the  total  population 
is  divided:  a  table  showing  the  probability  of  occurrence 
of  each  possible  value. 

PRODUCER'S  RISK:  The  probability  of  rejecting  an  item  which 
is,  in  fact,  satisfactory. 

PROGRAM:  (1)  A  plan  or  scheme  of  action  designed  for  the 

accomplishment  of  a  definite  objective  that  is  specific 
as  to  the  time-phasing  of  the  work  to  be  done  and  the 
means  proposed  for  its  accomplishment,  particularly  in 
quantitative  terms,  with  respect  to  manpower,  material, 
and  facilities  requirements:  thus  a  program  provides  a 
basis  for  budgeting:  (2)  a  segment  or  element  of  a 
complete  plan:  (3)  a  budget  account  classification. 
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PROGRAMMING:  The  process  of  translating  planned-military  force 

requirements  into  specific  time -phased ,  scheduled  actions, 
and  of  identifying  in  relatively  precise  terms  the 
resources  required.  It  is  the  bridge  between  planning 
and  budgeting. 

QUANTIFY:  To  qualify  with  respect  to  quantity.  In  analysis, 

to  translate  observed  physical  relationships  into  analogous 
mathematical  relationships. 

QUEUING  THEORY:  A  theory  that  deals  with  the  analysis  of  costs 
and  effectiveness  when  items  appear  with  some  randomness 
for  processing  at  a  facility  with  a  capacity  for  processing 
simultaneously  fewer  items  that  may  be  waiting  at  a 
given  time.  The  costs  are  costs  of  waiting  and  of  providing 
the  capacity  to  reduce  the  amount  of  waiting.  Examples 
of  queuing  problems  are:  (1)  determination  of  a  number 

of  checkout  counters  at  a  supermarket  that  minimizes 
the  sum  of  costs  of  customer  dissatisfaction  if  they  must 
wait  in  line  and  costs  of  providing  additional  checkers; 

(2)  determination  of  the  capacity  of  communications 
capacity  and  of  delays  in  the  processing  of  messages. 

RANDOM  ACCESS :  Pertaining  to  the  process  of  obtaining 

information  from  or  placing  information  into  computer 
storage  where  the  time  required  for  such  access  is 
independent  of  the  location  of  the  information  most 
recently  obtained  or  placed  in  storage. 

RANDOM  NUMBERS:  A  sequence  of  digits  in  which  each  digit  has 
an  equal  probability  of  occurring  in  each  position, 
wholly  independent  of  which  digits  appear  elsewhere  in 
the  sequence. 

RANDOM  NUMBER  GENERATOR:  A  special  computer  routine  designed 
to  produce  a  random  number  or  series  of  random  numbers 
according  to  specified  limitations. 

RANDOM  SAMPLE:  A  sam  pie  selected,  from  a  population  to  be 

tested,  in  such  a  manner  that  every  element  in  the  popu¬ 
lation  has  an  equal  chance  of  being  chosen  for  the  sample. 

RANDOM  VARIABLE:  A  function  defined  on  a  sample  space. 

It  is  called  discrete  if  it  assumes  only  a  finite 
or  denumerable  number  of  values  and  continuous  if  it 
assumes  a  continuum  of  values. 

R  CHARTS:  Charts  of  the  range  of  small  samples,  useful,  in 
monitoring  change  in  dispersion  in  the  product 
of  a  system. 
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REAL  TIME  OPERATION:  The  use  of  the  computer  as  an  element 

of  a  processing  system  in  which  the  times  of  occurrence 
of  data  transmission  are  controlled  by  other  portions 
of  the  system,  or  by  physical  events  outside  the  system, 
and  cannot  be  modified  for  convenience  in  computer 
programming. 

REDUNDANCY:  The  existence  of  more  than  one  means  for 

accomplishing  a  given  task,  where  all  means  must  fail 
before  there  is  an  overall  failure  of  the  system. 

RELATIVE  FREQUENCY:  The  ratio  of  the  number  of  observations 
(elements)  in  a  class  (subset)  to  the  total  number 
of  observations  constituting  a  population  (universe  or 
set). 

RELIABILITY:  The  probability  that  the  system  will  perform 

satisfactorily  for  at  least  a  given  period  of  time 
when  used  under  stated  conditions. 

REORDER  LEVEL:  The  inventory  balance  at  which  a  replacement 
order  is  placed. 

REPAIRABILITY :  The  probability  that  a  failed  system  will  be 

restored  to  operable  condition  within  a  specified 
active  repair  time. 

REPROGRAMMING:  The  reapplication  of  funds  between  budget 

activities  or  line  items  within  a  single  appropri  ati  on 
account. 

REQUIREMENT:  The  need  or  demand  for  personnel,  equipment, 

facilities,  other  resources  or  services,  expressed  in 
specific  quantities  for  specific  time  periods. 

RESEARCH  AND  DEVELOPMENT  (R&D)  :  Basic  and  applied  research 
in  the  sciences  and  engineering,  and  the  design  and 
development  of  prototypes  and  processes.  Excludes 
routine  product  testing,  market  research,  sales  promotion, 
sales  service,  and  other  non-technological  activites  or 
technical  services. 

Basic  research  includes  original  investigations  for 
the  advancement  of  scientific  knowledge  that  do  not  have 
specific  practical  objectives. 

Applied  research  is  the  practical  application  of  knowledge, 
material  and/or  techniques  toward  a  solution  to  an 
existent  or  anticipated  military  or  te chn o  1  ogi cal 
requirement . 
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Development  includes  technical  activities  of  a  nonroutine' 
nature  concerned  with  translating  research  findings  or 
other  scientific  knowledge  into  products  or  processes. 
Development  does  not  include  routine  technical  services 
or  other  activities  excluded  from  the  above  definition 
of  research  and  development. 

RES.EARCH  AND  DEVELOPMENT  (R&D)  COSTS:  The  cost  of 

developing  a  new  capability  to  the  point  where  it  is 
ready  for  procurement  for  operational  units. 

RESOURCE  IMPACT:  The  cost  of  adopting  a  course  of  action  stated 
i  n  measurable  terms.  Resource  impacts  cannot  always  be 
reduced  to  dollar  terms. 

REVOLVING  FUND:  A  fund  established  to  finance  a  cycle  of 

operations  to  which  reimbursements  and  collections  are 
returned  for  reuse  in  a  manner  such  as  to  maintain  the 
principle  of  the  fund;  e.g.,  working- capital  fund, 
stock  fund. 

RISK:  As  used  in  cost-effectiveness  analysis  and  operations 

research,  a  situation  is  characterized  as  risk  if  it  is 
possible  to  describe  all  possible  outcomes  and  to 
assign  meaningful,  objective  numerical  probability  weights 
to  each  one.  For  example,  an  action  might  lead  to  this 
risky  outcome:  a  reward  of  $10  if  a  "fair"  coin  comes 
up  heads,  and  a  loss  of  $5  1  f  it  comes  up  tails. 

Another  example,  50%  of  all  missiles  fired  can  be  expected 
to  land  within  one  CEP  of  the  target. 

ROUTINE:  A  set  of  coded  instructions  arranged  in  proper  sequence 

to  direct  the  computer  to  perform  a  desired  operation 
or  sequence  of  operations. 

SAMPLE  SPACE:  The  range  of  feasible  solutions. 

SAMPLING:  The  process  of  determining  characteristics  of  a 

population  by  collecting  and  analyzing  data  from  a 
representative  segment  of  the  population. 

SAMPLING  ERROR:  That  part  of  the  variation  in  the  data 

resulting  from  an  experiment  that  is  not  explained  by 
the  variation  in  the  factors  controlled  during 
the  experimentation. 

SCENARIO:  A  word  picture  of  a  fixed  sequence  of  events  in  a 

defined  environment. 
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SENSITIVITY  ANALYSIS:  Repetition  of  an  analysis  with  different 
quantitative  values  for  cost  or  operational  assumptions 
or  estimates  such  as  h  i  t  -  k  i  1 1  p  r  o  b  ab  i  1  i  t  i  e  s,  activity 
rates,  or  R&D  costs,  in  order  to  determine  their  effects 
for  the  purposes  of  comparison  with  the  results  of  the 
basic  analysis.  If  a  small  change  in  an  assumption 
results  in  a  proportionately  or  greater  change  in  the 
the  results,  then  the  results  are  said  to  be  sensitive 
to  that  assumption  or  parameter. 

SETS:  A  collection  of  items  (elements)  chosen  as  pertinent. 

SHADOW  PRICE:  The  shadow  price  of  a  factor  is  a  measure  of  its 
opportunity  cost  or  its  marginal  product.  For  example, 
when  unemployment  is  widespread,  the  opportunity  cost 
of  labor  may  be  near  zero,  so  that  the  shadow  price  of 
labor  may  be  well  below  the  prevailing  wages  of 
those  workers  who  are  actually  employed. 

SIMULATION:  The  representation  of  physical  systems  and 

phenomena  by  computers,  models,  or  other  equipment. 

The  model  or  computer  representation  is  manipulated  to 
imitate  significant  aspects  of  a  situation, 

SPURIOUS  CORRELATION:  Accidental  correlation  having  no 

causative  basis  and  without  expectation  of  continuance. 

STANDARD  DEVIATION:  A  measure  of  the  dispersion  of  observed 
data.  Mathematically,  it  is  the  positive  square  root 
of  the  variance. 

STANDARD  ERROR:  The  standard  deviation  of  a  group  of  measures 
of  the  same  characteristics  (often  termed  a  "statistic" 
or  a  "parameter"),  each  obtained  from  a  distinct  sample 
drawn  from  a  larger  "universe"  or  "population"  . 

STATISTICAL  BIAS:  If  some  samples  or  observation  data  are 

more  likely  to  be  chosen  than  others,  or  if  subjective 
methods  are  used  in  selecting  sample  data,  the  results 
are  considered  biased. 

STATISTICAL  DECISION  THEORY:  Theory  dealing  with  logical 

analysis  of  choice  among  courses  of  action  when  (l)the 
consequence  of  any  course  of  action  will  depend  upon  the 
"state  of  the  world",  (2)  the  true  state  is  as  yet  unknown, 
but  (3)  it  is  possible  at  a  cost  to  obtain  additional 
information  about  the  state. 
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STOCHASTIC  PROCESS:  The  statistical  concept  underlying  the 
prediction  of  the  condition  of  an  element  of  a 
larger  group  when  the  probable  average  condition 
of  tine  larger  group  is  known.  For  example,  assume  that 
an  armored  division,  under  certain  circumstances,  has 
on  the  average  a  certain  number  of  tanks  deadlined  for 
unscheduled  maintenance .  The  probability  that 
any  given  tank  under  the  same  circumstances  will  be  dead- 
lined  for  unscheduled  maintenance  on  a  specific  day  is 
described  by  a  stochastic  process. 

STOCKOUT  COST:  The  cost  due  to  disrupted  schedules  or  to 
inability  to  satisfy  customers  because  items 
ordinarily  stocked  are  not  available, 

STORAGE  TIME:  The  time  during  which  the  system  is 

presumed  to  be  inoperable  condition,  but  is  being  held 
for  emergency;  e.g,,  as  a  spare. 

SUBOPTIMIZATION:  Optimization  refers  to  a  selection  of  a  set 

of  actions  that  maximize  the  achievement  of  some  objective 
subject  to  all  of  the  real  constraints  that  exist. 

For  example,  one  may  optimize  a  choice  of  weapons  for 
achieving  certain  objectives  of  a  decision  but  within 
the  given  constraint  of  a  certain  maximum  cost  of  a 
division.  But  one  suboptimizes  on  achievement  of  the 
division  objective  if  he  is  given  discretion  only  over 
the  amount  and  kind  of  armor  and  is  given  a  maximum 
amount  of  money  to  spend  on  armor.  The  objective  he 
maximizes  directly  may  be  only  the  mission  of  armor  in 
the  division's  objective.  Such  a  suboptimization  will 
yield  something  inferior  to  an  optimized  expenditure 
on  different  kinds  of  armor  if  the  total  budget  for 
armor  given  to  the  suboptimizer  is  really  not  optimal, 
or  if  there  are  interdependencies  between  decisions  on 
armor  and  decisions  on  other  things  that  are  outside  the 
discretion  of  the  person  suboptimizing  on  armor. 

SUBROUTINE:  The  set  of  instructions  necessary  to  direct 

the  computer  to  carry  out  a  well  defined  mathematical 
or  logical  operation. 

SUBSET:  A  collection  wholly  contained  within  a  larger  collection 
a  group  of  elements  constituting  part  of  a  universe. 

SYMBOLIC  LOGIC:  The  study  of  formal  logic  and  mathematics 

by  means  of  a  special  written  language  which  seeks  to 
avoid  the  ambiguity  and  inadequacy  of  ordinary  language. 

SYSTEM:  Weapon  system  is  composed  of  equipment,  skills  and 

techniques ,  the  composite  of  which  forms  an  instrument 
of  combat.  The  complete  weapon  system  includes  all 
related  facilities,  equipment,  materials,  services,  and 
personnel  required  solely  for  its  operation,  so  that  the 
instrument  of  combat  becomes  a  self-sufficient  unit 
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of  striking  power  in  its  intended  operational  environment. 
Support  system  is  a  composite  of  equipment,  skills,  and 
techniques  that,  while  not  an  instrument  of  combat,  is 
capable  of  performing  a  clearly  defined  function  in  support 
of  a  mission.  A  complete  support  system  includes  all 
related  facilities,  equipment,  materials,  services,  and 
personnel  required  for  operation  of  the  system,  so  that 
it  can  be  considered  a  self-sufficient  unit  in  its  intended 
operational  environment. 

SYSTEMS  ANALYSIS  (SA)  :  A  formal  inquiry  intended  to  advise 

a  decision  maker  on  the  policy  choices  involved  in  a  major 
decisions.  In  DoD  a  systems  analysis  may  be  concerned 
with  such  matters  as  weapon  development,  force  posture 
design,  or  the  determination  of  strategic  objectives. 

To  qualify  as  a  system  analysis  a  study  must  look  at 
an  entire  problem  as  a  whole.  Characteristically,  it 
will  involve  a  systematic  investigation  of  the  decision¬ 
maker's  objectives  and  of  the  relevant  criteria; 
a  comparison--quantitative  when  possible--of  the  costs, 
effectiveness,  and  risks  associated  with  the  alternative 
policies  or  strategies  for  achieving  each  objective;  and 
an  attempt  to  formulate  additional  alternatives  if  those 
examined  are  deficient. 

SYSTEMS  EFFECTIVENESS:  The  probability  that  the  system  can 
successfully  meet  an  operational  demand  within  a  given 
time  when  operated  under  specified  conditions. 

SYSTEMS  APPROACH:  The  art  of  examining  the  entire  context  within  which 
the  iten  of  interest  will  function. 

TCHEBYCHEFF ' S  THEOREM:  The  proportion  of  the  observations 
falling  between  -ko  and  +ko  is  at  least  as  large  as 
l-(l/k2)  regardless  of  the  distribution. 

TIME-PHASED  COSTS:  A  presentation  of  the  cost  results 
broken  down  by  the  time  period  in  which  the  costs 
occur  rather  than  a  single  total  cost  figure. 

TOTAL  OBLIGATION  AUTHORITY (TOA) :  The  cost  allocated  to  a 

given  system  or  organization.  This  cost  when  related 

to  a  specific  time  period,  for  example  a  year,  represents 

obligations  that  can  be  incurred  during  that  year  and 

not  necessarily  expenditures.  The  total  obligation  authority 

for  a  specific  year  to  furnish  a  house  is  the  cost  of 

what  can  be  ordered  during  that  year  even  if  deliveries 

and  payments  are  made  in  later  years. 

TOTAL  SYSTEM  COST:  The  total  R&D,  Investment,  and  Operating 
Costs  (fora  specified  number  of  years  of  operation) 
required  to  develop,  procure,  and  operate  the  particular 
weapon  system. 
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TRANSFORM:  The  derivation  of  a  new  body  of  data  from  a  given 

one  according  to  specific  procedures,  often  leaving  some 
feature  invariant. 

TROOP  TEST:  A  troop  test  is  a  test  conducted  in  the  field, 
using  TOE  units  or  units  organized  under  proposed  TOE, 
to  evaluate  current  or  proposed  doctrine  and  organizations. 
Material  is  considered  in  the  conduct  of  troop  tests 
only  insofar  as  material  affects  the  doctrine  or 
organization  being  evaluated. 

TYPE  I  ERROR:  The  belief  that  something  true  is  false. 

TYPE  II  ERROR:  The  belief  that  something  false  is  true. 

UNCERTAINTY:  A  situation  is  uncertain  if  there  is  no  objective 

basis  for  assigning  numerical  probability  weights  to  the 
different  possible  outcomes  or  there  is  no  way  to  describe 
the  possible  outcomes.  For  example,  the  probability 
of  a  foreign  nation  continuing  to  furnish  the  U.S.  with 
base  rights  is  an  uncertainty 

UTILITY:  a  personal  subjective  value  of  a  tangible  or 
intangible  commodity. 

VALUE  ADDED  BY  MANUFACTURE:  That  part  of  value  given  products 
shipped  actually  created  within  a  given  industry.  The 
unadjusted  series  is  calculated  by  subtracting  the  cost 
of  materials,  supplies,  containers,  fuel,  purchased  electric 
energy,  and  contract  work  from  the  value  of  shipments. 

The  adjusted  series,  which  is  more  inclusive,  is  equal 
to  the  unadjusted  series  plus:  (1)  value  added  by 
merchandising  operations,  and  (2)  the  net  change  in 
inventories  (both  finished  goods  and  work-in-progress) 
between  the  beginning  and  end  of  the  year.  (Value  added 
is  almost  free  statistically  from  the  duplication  of  values 
existing  in  the  value  of  shipments  and  approximates  the 
net  value  of  manufacturers). 

VARIABLE  COSTS:  Those  costs  that  vary  with  the  volume  of  output 
as  contrasted  with  fixed  costs,  which  do  not  vary  with 
output. 

VARIABLES:  General  numbers,  such  as  x  or  y  which  may  take  on 

many  values  or  which  may  have  conditional  fixed  values 
as  in  +  2x  =  19 . 

VARIANCE:  A  measure  of  dispersion  of  a  frequency  distribution 

computed  by  summing  the  squares  of  the  difference 
between  each  observation  and  the  arithmetic  mean  of  the 
distribution  and  then  dividing  by  the  number  of 
observations . 
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WAR  GAME:  A  simulation,  by  whatever  means,  of  a  military 
operation  involving  two  or  more  opposing  forces, 
conducted  using  rules,  data,  and  procedures 
designated  to  depict  an  actual  or  assumed  real  life 
situation . 
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0.4236 

3.4251 

0.42® 

0.4279 

0.4292 

0.4306 

0.4319 

1.5 

0.4332 

1.  4345 

0.4357 

0.4370 

3.4382 

0.4394 

0.4406 

0.4418 

C 

.4430 

0.4441 

1.6 

0.4452 

3 , 4463 

0.4474 

0.4485 

3.4495 

0.4505 

0.4515 

0.4525 

0.4535 

0.4545 

1.7 

0.4554 

1.4564 

0.4573 

0.4582 

3.4591 

0.4599 

0.4608 

0.4616 

0.4625 

0,4633 

1.8 

0.4641 

1.4649 

0.4656 

0.4664 

3.4671 

0.4678 

0.4686 

0.4693 

0.4699 

0.4706 

1.9 

0.4713 

1.  4719 

0.4726 

0.4732 

3.4738 

0.4744 

0.4750 

0.4758 

0.4762 

0.4767 

2.0 

0.4773 

I.  4778 

0.4783 

0.4788 

3.4793 

0.4798 

0.4803 

0.4808 

c 

.4812 

0.4817 

2.1 

0.4821 

1.4826 

0.4830 

0.4834 

3.4838 

0.4842 

0.4846 

0.4850 

0.4854 

0.4857 

2.2 

0.4861 

3.4865 

0.4868 

0.4871 

3.4875 

0.4878 

0.4881 

0.4884 

0.4887 

0.4890 

2.3 

0.4893 

'  .4896 

0.4898 

0.4901 

3.4904 

0.4906 

0.4909 

0.4911 

0.4913 

0.4916 

2.4 

0.4918 

I.  4920 

O.4922 

0.4925 

3.4927 

O.4929 

0.4931 

0.4932 

C 

.4934 

0.4936 

2.5 

0.4938 

L  4940 

0.4941 

0.4943 

3.4945 

0.4946 

0.4948 

0.4949 

0.4951 

0.4952 

26 

0 . 4953 

1.4955 

0.4956 

0.4957 

3.4959 

0.4960 

0.4961 

0.4962 

0.4963 

0.4964 

2.7 

0.4965 

3,4966 

0.4967 

0.4968 

3.4969 

0.4970 

0.4971 

0.4972 

0.4973 

0.4974 

2.8 

0.4974 

1.4975 

0.4976 

0.4977 

3.4977 

0.4978 

0.4979 

0.4980 

.4980 

0.4981 

2.9 

0.4981 

1.4982 

0.4983 

0.4984 

3.4984 

0.4984 

0.4985 

0.4985 

0.4986 

0.4986 

3.0 

0 , 4986; 

1.4987 

0.4987 

0.4988 

3.4968 

3.4988 

0.4989 

0.4989 

0.4989 

0,4990 

3.1 

0,49902 

1.4991 

0.4991 

0.4991 

3.4992 

0 . 4992 

0.4992 

0.4992 

0.4993 

0.4993 

3.2 

0 . 4993  J 

3.3 

0.49951 

3.4 

0.49966 

3.5 

0.49977 

3.6 

0.49981 

3.7 

0.49985 

3.8 

0,4999! 

3.9 

0.49995 

4.0 

0,4999? 

Area  *  j 

z 

f(0 

0 

dz '  =  r 

dz 

0 

f(0 

dz‘ 

c-2 
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TABLE  C-3 

DISTRIBUTION  OF  X; 


Probabi  1  i  ty.  p> 


0.99 

0.975 

0.95 

0.90 

0.80 

0.75 

0.50 

0.25 

0.20 

O.IO 

000157 

000982 

0.00393 

0.0158 

0.0642 

0.10153 

0.455 

1.323 

1.642 

2.706 

0.0201 

0.0506 

0.103 

0211 

0.446 

0.5753 

1.386 

2.772 

3.219 

4.605 

0.115 

0.216 

0.352 

0.584 

1.005 

1.2125 

2.366 

4.108 

4.642 

6.251 

0.297 

0.484 

0.711 

1.064 

1.649 

1.9225 

3-357 

5-385 

5.989 

7.779 

0.554 

0.831 

1.145 

1.610 

2.343 

2.674 

4.351 

6.625 

7.289 

9.236 

0.872 

1.237 

1.635 

2.204 

3.070 

3.454 

5.348 

7.840 

8.558 

10.645 

1-239 

1.689 

2.167 

2.833 

3.822 

4.254 

6.346 

9.037 

9.803 

12.017 

1.646 

2.179 

2.733 

3-490 

4.594 

5.070 

7.344 

10.218 

11.030 

13.362 

2.088 

2.700 

3.325 

4.168 

5.380 

5.898 

8.343 

11.388 

12.242 

14.684 

2.558 

3.247 

3-94o 

4.865 

6.179 

6.737 

9.342 

12.548 

13.442 

15.987 

3.053 

3.816 

4-575 

5.578 

6.989 

7.584 

10.341 

13.701 

14.631 

17.275 

3.571 

4.404 

5.226 

6.304 

7.807 

8.438 

11.340 

14.845 

15.812 

18.549 

4.107 

5.008 

5.892 

7.042 

8.634 

. 9-299 

12.340 

15.984 

16.985 

19.812 

4.660 

5.628 

6.571 

7.790 

9.467 

10.165 

13.339 

17.117 

18.151 

21.064 

5.229 

6.262 

7.261 

8,547 

LO.  307 

11.036 

14.339 

18.245 

19.311 

22.307 

5.812 

6.907 

7.962 

9-312 

LI. 152 

11.912 

15-338 

19 . 368 

20.465 

23.542 

6.408 

7.564 

8.672 

10 . 085 

L2.002 

12.791 

16.338 

lo .  488 

21.615 

24.769 

7-015 

8.231 

9-390 

10 .  865 

L2.857 

13.675 

17.338 

?l . 605 

22.760 

25.989 

7.633 

8.906 

lo.117 

11.651 

-3.716 

14.562 

18.338 

22.717 

23.900 

27.204 

8.260 

9.591 

to.  851 

12.443 

-4.578 

15.452 

19.337 

23.827 

25.038 

28.412 

8.857 

.0 . 283 

Ll-591 

13  -  240 

L5.445 

16.344 

20 . 337 

?4.935 

26.171 

29.615 

9.5  2 

.0.982 

t2 .338 

14.041 

L6.314 

17.239 

21.337 

26.039 

27.301 

30.813 

.0.196 

.1.688 

L3.091 

14.848 

-7.187 

18.137 

22.337 

27.141 

28.429 

32.007 

.0.856 

.2.400 

L3.848 

15-659 

-8.062 

19.037 

23.337 

?8.24l 

29.553 

33.196 

_  *24 

-3.119 

t4.61 1 

16.473 

L8.940 

19.939 

24.337 

59.339 

30.675 

34.382 

.2.198 

-3.844 

L5-379 

17.292 

19 . 820 

20.843 

25-336 

30.434 

31.795 

35.563 

.2.879 

-4.573 

L6.151 

L8 . 114 

20.703 

21.749 

26.336 

31.528 

32.912 

36.741 

3.565 

-5.308 

l6 . 928 

18.933 

21.588 

22.657 

27.336 

32 . 620 

34.027 

37.916 

.4 . 256 

.6.047 

-7.708 

L9.768 

22.475 

23.566 

28.336 

33.711 

35.139 

39.087 

.4.953 

.6.791 

-8.493 

20.599 

23.364 

24.476 

29.336 

34-799 

36.250 

to. 256 

3.841 

5.991 

7.815 

9.488 

11.070 

12.592 

14.067 

15.507 

16.919 

18.307 


26 . 296 
27.587 
28.869 
30.144 
31.410 

32.671 

33.924 

35.172 

36.415 

37.652 

38.885 

40.113 

H.337 

^2.557 

^3.773 


5.024 

7-377 

9.348 

11.143 

12.832 

14.449 

16.013 

17.534 

19.023 

20.483 


21.920 

23-336 

24.735 

26.119 

27.488 


28.845 

30.191 

31.526 

32.852 

34.169 

35.479 

36.780 

38.075 

39-364 

5-0.646 

41.923 
43.194 
44.460 
45.722 
46.  96O 


For  degrees  of  freedom  greater  than  30, 
mean  \/2(DF)-i  and  variance  1. 


the  quantity  \/2x^  is  approximately  normally  distributed  with 


O.Ol 

0.001 

6.635 

10.827 

9.210 

13.815 

11.345 

16.268 

18.465 

20.517 

16.812 

22.457 

18.475 

24.322 

20.090 

26.125 

21.666 

23.209 

27 . 877 

29.588 

24.725 

31.264 

26.217 

32.909 

27.688 

34.528 

29.141 

36.123 

30.578 

37-697 

32. 000 

39.252 

33.^09 

40.790 

34.805 

42.312 

36.191 

43 . 820 

37.566 

45.315 

38.932 

46.797 

40.289 

48.268 

41.638 

49.728 

42.980 

44.314 

51.179 

52.620 

45.642 

54.052 

46.963 

55.476 

48.278 

56.893 

49.588 

58.302 

50.892 

59.703 
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TABLE  C-4 


F  Distribution:  Upper  l  Percent  Points 


Degrees  of  Freedom  In  Numerator  % 


1 

2 

3 

4 

c, 

6 

7 

8 

9 

1 

4052.2 

4999.5 

5403.3 

5624.6 

5763.7 

5859.0 

5928.3 

5981.6 

6022.5 

2 

98.503 

99.000 

99.166 

99.249 

99.299 

99.332 

99.356 

99.374 

99.388 

3 

34.116 

30.817 

29.457 

28.710 

28.237 

27.911 

27 . 672 

27.489 

27 . 345 

4 

21.198 

18.000 

16.694 

15.977 

15.522 

15.207 

14.976 

14.799 

14.659 

5 

16.258 

13.274 

12.060 

11.392 

io. 967 

10.672 

10.456 

10.289 

10.158 

6 

13.745 

10.925 

9.7795 

9.1483 

8.7459 

8.4661 

8.2600 

8.1016 

7.9761 

7 

12.246 

9.5466 

8.4513 

7.8467 

7.4604 

7.1914 

6.9928 

6.8401 

6.7188 

8 

11.259 

a.  6491 

7.5910 

7,0060 

6.6318 

6.3707 

6.1776 

6.0289 

5.9106 

9 

10.561 

8.0215 

6.9919 

6.4221 

6.0569 

5.8018 

5.6129 

5.4671 

5.35H 

10 

10.044 

7.5594 

6.5523 

5.9943 

5.6363 

5.3858 

5.2001 

5.0567 

4.9424 

11 

9.6460 

7.2057 

6.2167 

5.6683 

5.3160 

5.0692 

4.8861 

4.7445 

4.6315 

12 

9.3302 

6.9266 

5.9526 

5.4119 

5.0643 

4.8206 

4.6395 

4.4994 

4.3875 

13 

9.0738 

6.7010 

5.7394 

5.2053 

4.8616 

4.6204 

4.4410 

4.3021 

4.1911 

14 

8.8616 

6.5149 

5.5639 

5.0354 

4.6950 

4.4558 

4.2779 

4.1399 

4.0297 

15 

8.6831 

6.3589 

5.4170 

4.8932 

4-5556 

4.3183 

4.1415 

4.0045 

3.8948 

16 

8.5310 

6.2262 

5.2922 

4.7726 

4.4374 

4.2016 

4.0259 

3.8896 

3.7804 

17 

8.3997 

6.1121 

5.1850 

4.6690 

4.3359 

4.1015 

3.9267 

3.7910 

3.6822 

18 

8.2854 

6.0129 

5.0919 

4.5790 

4.2479 

4.0146 

3.8406 

3.7054 

3.5971 

19 

8.1850 

5.9259 

5.0103 

4.5003 

4.1708 

3.9386 

3.7653 

3.6305 

3.5225 

20 

8.0960 

5.8489 

4.9382 

4.4307 

4 . 1027 

3.8714 

3.6987 

3.5644 

3.4567 

21 

8.0166 

5.7804 

4 . 8740 

4.3688 

4.0421 

3.8117 

3.6396 

3.5056 

3-3961 

22 

7.9454 

5.7190 

4.8166 

4.3134 

3.9880 

3.7583 

3.5867 

3.4530 

3-3458 

23 

7.8811 

5.6637 

4.7649 

4.2635 

3.9392 

3.7102 

3.5290 

3.4057 

3.2986 

24 

7.8229 

5.6136 

4.7181 

4.2184 

3.8951 

3.6667 

3.4959 

3.3629 

3.2560 

25 

7.7698 

5.5680 

4.6755 

4.1774 

3.8550 

3.6272 

3.4568 

3-3239 

3.2172 

26 

7.7213 

5.5263 

4.6366 

4.1400 

3.8183 

3.59H 

3.4210 

3.2884 

3.1818 

27 

7.6767 

5.4881 

4 . 0009 

4.1056 

3.7848 

3.5580 

3.3882 

3.2558 

3.1494 

28 

7.6356 

5.4529 

4.5681 

4.0740 

3.7539 

3.5276 

3.3581 

3.2259 

3.H95 

29 

7.5976 

5.4205 

4.5378 

4.0449 

3.7254 

3.4995 

3.3302 

3.1982 

3.0920 

30 

7.5625 

5.3904 

4.5097 

4.0179 

3.6990 

3.4735 

3.3045 

3.1726 

3.0665 

40 

7.3141 

5.1785 

4.3126 

3.8283 

3.5138 

3.2910 

3.1238 

2.9930 

2.8876 

60 

7.0771 

4.9774 

4.1259 

3.6491 

3.3389 

3.1187 

2.9530 

2.8233 

2.7185 

120 

6.8510 

4.7865 

3-9493 

3.4796 

3. 1735 

2.9559 

2.7918 

2.6629 

2.5586 

00 

6.6349 

4.6052 

3.7816 

3.3192 

3.0173 

2.8020 

2 . 6393 

2.5113 

2.4073 
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_ TABLE  C-4  (continued) 

F  Distribution:  Upper  1  Percent  Points 
Degrees  of  Freedom  i  n  Numerator 


10 

12 

15 

20 

24 

30 

40 

60 

120 

00 

1 

6055.8 

6106.3 

6157.3 

6208.7 

6234.6 

6260.7 

6286.8 

6313.0 

6339. 4 

6366.0 

2 

99-  399 

99.416 

99-  432 

99-  449 

99. 458 

99.466 

99.474 

99 . 483 

99.491 

99.501 

3 

27.229 

27.052 

26.872 

26.690 

26.598 

26,505 

26.411 

26.316 

26.221 

26.125 

4 

14.546 

14.374 

14.198 

14.020 

13-929 

13.838 

13.745 

13.652 

13-558 

13.463 

5 

io.051 

9.8883 

9.7222 

9- 5527 

9.4665 

9.  3793 

9.2912 

9.2020 

9.1118 

9.0204 

6 

7.8741 

7.7183 

7.  5590 

7-3958 

7.  3127 

7.2285 

7.1432 

7.0568 

6.9690 

6.8801 

7 

6.6201 

6.4691 

6.3143 

6,1554 

6.0743 

5.9921 

5.9084 

5.8326 

5.7572 

5.  6495 

8 

5.8143 

5 . 6668 

5.5151 

5.  3591 

5. 2793 

5. 1981 

5.1156 

5.0316 

4.9460 

4.8588 

9 

5.2565 

5.1114 

4,9621 

4.8080 

4,7290 

4.6486 

4.5667 

4,4831 

4.  3978 

4.3105 

10 

4.  8492 

4. 7059 

4.5582 

4.4054 

4.3269 

4,2469 

4.1653 

4.0819 

3. 9965 

3.9090 

11 

4.  5393 

4.  3974 

4.2509 

4,0990 

4.0209 

3.9411 

3.8596 

3.7761 

3.6904 

3.6025 

12 

4.2961 

4. 1553 

4.0096 

3.8584 

3,7805 

3.7008 

3.6192 

3.  5355 

3.4494 

3.3608 

13 

4.1003 

3.9603 

3.8154 

3.6646 

3.5868 

3.5070 

3.  4253 

3.  3413 

3.2548 

3.1654 

14 

3.  9394 

3.8001 

3.  6557 

3.5052 

3.4274 

3.  3476 

3.2656 

3.1813 

3.0942 

3.0040 

15 

3.8049 

3,6662 

3.5222 

3.  3719 

3.2940 

3.2141 

3.1319 

3,0471 

2 . 9595 

2.8684 

16 

3.6909 

3.  5527 

3.4089 

3.2588 

3.1808 

3.1007 

3.0182 

2  •  9330 

2,8447 

2.7528 

17 

3.  5931 

3.4552 

3.3117 

3.1615 

3.0835 

3.0032 

2.9205 

2.8348 

2 . 7459 

2.6530 

18 

3.5082 

3.3706 

3.2273 

3,0771 

2.9990 

2.9185 

2.8354 

2.7493 

2.  6597 

2.5660 

19 

3.4338 

3.2965 

3. 1533 

3.0031 

2,9249 

2,8442 

2.7608 

2.6742 

2 . 5839 

2.4893 

20 

3.  3682 

3.2311 

3.0880 

2 . 9377 

2,8594 

2 . 7785 

2.6947 

2.6077 

2.5168 

2.4212 

21 

3. 3098 

3.1729' 

3.0299 

2.8796 

2.8011 

2.7200 

2.  6359 

2.5484 

2.4568 

2.3603 

22 

3.2576 

3.1209 

2.9780 

2.8274 

2.7488 

2.6675 

2.5831 

2,4951 

2.4029 

2 . 3055 

23 

3.2106 

3.0740 

2.9311 

2.7805 

2.7017 

2.6202 

2 . 5355 

2.4471 

2.3542 

2.2559 

24 

3.1681 

3.0316 

2.8887 

2.7380 

2.6591 

2 . 5773 

2.4923 

2.4035 

2 . 3099 

2.2107 

25 

3.1294 

2.9931 

2.8502 

2 . 6993 

2.6203 

2 . 5383 

2.4530 

2.3637 

2.2695 

2.1694 

26 

3.0941 

2 . 9579 

2.8150 

2.6640 

2.5848 

2.5026 

2.4170 

2. 3273 

2.2325 

2.1315 

27 

3.0618 

2.9256 

2.7827 

2.6316 

2.5522 

2 , 4699 

2.3840 

2.2938 

2.1984 

2.0965 

28 

3.0320 

2 . 8959 

2,7530 

2.6017 

2.5223 

2.4397 

2.3535 

2.2629 

2.1670 

2.0642 

29 

3.0045 

2.8685 

2.7256 

2 , 5742 

2.4946 

2.4118 

2,3253 

2.2344 

2.1378 

2.0342 

30 

2 . 9791 

2,8431 

2.7002 

2.5487 

2,4689 

2.3860 

2.2992 

2.2079 

2.1107 

2.0062 

40 

2.8005 

2.6648 

2.5216 

2.3689 

2.2880 

■2.2034 

2.1142 

2.0194 

1.9172 

1 . 8047 

60 

2.6318 

2.4961 

2 . 3523 

2.1978 

2.1154 

2.0285 

1.9360 

1.8363 

1,7263 

1.6006 

120 

2.4721 

2.3363 

2.1915 

2.0346 

1 . 9500 

1.8600 

1,7628 

1 . 6557 

1 . 5330 

1 , 3805 

00 

2 . 3209 

2,1848 

2.0385 

1.8783 

1.7908 

1.6964 

1  •  5923 

1.4730 

1.  3246 

1.000 
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TABLE  C-4  (continued) 

F  Distribution:  Upper  2.5  Percent  Points 


Degrees  of  Freedom  in  Numerator 


1 

2 

3 

4 

5 

6 

7 

8 

9 

647.79 

799.50 

864.16 

899.58 

921.85 

937.11 

948.22 

956.66 

963.28 

38.506 

39,000 

39.165 

39.248 

39.298 

39-331 

39-355 

39.373 

39.387 

17.443 

16.044 

15.439 

15.101 

14 . 885 

14.735 

14.624 

14 . 540 

14.473 

12.218 

10 . 649 

9.9792 

9.6045 

9.3645 

9.1973 

9.0741 

8 . 9796 

8.9047 

10.007 

8.4336 

7.7636 

7.3879 

7.1464 

6.9777 

6.8531 

6.7572 

6.6810 

8.8131 

7.2598 

6.5988 

6.2272 

5.9876 

5.8197 

5-6955 

5-5996 

5.5234 

8.0727 

6.5415 

5.8898 

5.5226 

5.2852 

5.1186 

4-9949 

4.8994 

4.8232 

8 

7.5709 

6.0595 

5.4160 

5.0526 

4.8173 

4.6517 

4,5286 

4.4332 

4.3572 

9 

7.2093 

5-7147 

5.0781 

4.7181 

4.4844 

4-3197 

4.1971 

4,1020 

4.0260 

10 

6.9367 

5.4564 

4.8256 

4.4683 

4.2361 

4.0721 

3.9498 

3-8549 

3.7790 

11 

6.7241 

5-2559 

4.6300 

4.2751 

4.0440 

3.8807 

3.7586 

3.6638. 

3.5879 

12 

6.5538 

5-0959 

4.4742 

4.1212 

3.8911 

3.7283 

3.6065 

3.5118 

3.4358 

13 

6.4143 

4.9653 

4 . 3472 

3-9959 

3.7567 

3.6043 

3.4827 

3.3880 

3.3120 

14 

6.2979 

4.8567 

4.2417 

3.8919 

3.6634 

3.5014 

3.3799 

3.2853 

3.2093 

15 

6.1995 

4.7650 

4.1528 

3.8043 

3. 5764 

3.4147 

3.2934 

3.1987 

3.1227 

16 

6.1151 

4.6867 

4.0768 

3-7294 

3.5021 

3.3406 

3.2194 

3.1248 

3.0488 

17 

6.0420 

4.6189 

4.0112 

3.6648 

3e4379 

3.2767 

3.1556 

3.0610 

2.9849 

18 

5-9781 

4.5597 

3.9539 

3.6083 

3.3820 

3.2209 

3-0999 

3-0053 

2  .9291 

19 

5.9216 

4.5075 

3.9034 

3-5587 

3. 3327 

3.1718 

3.0509 

2 . 9563 

2.8800 

20 

5.8715 

4.4613 

3.8587 

3-5147 

3.2891 

3.1283 

3.0074 

2.9128 

2.8365 

2  1 

5.8266 

4.4199 

3.8188 

3.4754 

3.2501 

3.0895 

2.9686 

2.8740 

2 . 7977 

22 

5.7863 

4.3828 

3.7829 

3.4401 

3.2151 

3.0546 

2.9338 

2.8392 

2.7628 

23 

5.7498 

4.3492 

3.7505 

3.4083 

3.1835 

3.0232 

2.9024 

2 . 8077 

2.7313 

24 

5.7167 

4.3187 

3.7211 

3-3794 

3.1548 

2.9946 

2.8738 

2.7791 

2.7027 

25 

5.6864 

4.2909 

3.6943 

3.3530 

3.1287 

2.9685 

2.8478 

2.7531 

2.6766 

26 

5.6586 

4.2655 

3.6697 

3.3289 

3,1048 

2.9447 

2.8240 

2.7293 

2.6528 

27 

5.6331 

4.2421 

3.6472 

3.3067 

3.0828 

2.9228 

2.8021 

2.7074 

2.6309 

28 

5.6096 

4.2205 

3.6264 

3.2863 

3.0625 

2.9027 

2.7820 

2.6872 

2.6106 

29 

5.5878 

4.2006 

3.6072 

3.2674 

3.0438 

2.8840 

2.7633 

2.6686 

2.5919 

30 

5-5675 

4.1821 

3.5894 

3.2499 

3.0265 

2.8667 

2.7460 

2.6513 

2.5746 

40 

5.4239 

4.0510 

3.4633 

3.1261 

2.9037 

2.7444 

2.6238 

2.5289 

2.4519 

60 

5.2857 

3.9253 

3.3425 

3.0077 

2.7863 

2.5068 

2.4117 

2.3344 

120 

5.1524 

3,8046 

3.2270 

2.8943 

2.6740 

2.3948 

2.2994 

2.2217 

oo 

5.0239 

3.6889 

3.1161 

2.7858 

2.5665 

2.4082 

2.2875 

2.1918 

2.1136 
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_ TABLE  C-4  (continued) 

F  Distribution:  Upper  2.5  Percent  Points 


Degrees  of  Freedom  in  Numerator  v  ^ 


10 

12 

15 

20 

24 

30 

40 

60 

120 

ob 

1 

968.63 

976.71 

984.87 

993-10 

997.25 

1001.4 

1005.6 

1009.8 

1014.0 

E 

2 

39.398 

39.415 

39.431 

39.440 

39.456 

39.465 

39.473 

39.481 

39.490 

39.498 

3 

14.419 

14.337 

14.253 

14.167 

14.124 

14.081 

14.037 

13.992 

13.947 

4 

8.8439 

8.7512 

8.6565 

8.5599 

8.5109 

8.4613 

8.4111 

8.3604 

8.3092 

8.2573 

5 

6.6192 

6.5246 

6.4277 

6 . 3285 

6.2780 

6.2269 

6.1751 

6.1225 

6.0693 

6.0153 

6 

5.4613 

5.3662 

5.2687 

5,1684 

5.1172 

5.0652 

5.0125 

4.9589 

4.9045 

4.8491 

7 

4.7611 

4.6658 

4.5678 

4.466? 

4.4150 

4.3624 

4.3089 

4.2544 

4.1989 

4.1423 

8 

4.2951 

4. 1997 

4.1012 

3.9995 

3.9472 

3.8940 

3.8398 

3.7844 

3.7279 

9 

3.9639 

3.8682 

3.7694 

3.6669 

3.6142 

3.5604 

3.5055 

3.4493 

3.3918 

3.3329 

10 

3.7168 

3.6209 

3.5217 

3.4186 

3.3654 

3-3110 

3.2554 

3.1984 

3. 1399 

11 

3.5257 

3.4296 

3.3299 

3.2261 

3.1725 

3.1176 

3.0613 

3.0035 

2.9441 

2.8828 

12 

3.3736 

3.2773 

3.1772 

3.0728 

3.0187 

2.9633 

2.9063 

2.8478 

2.7874 

2.7249 

13 

3.2497 

3.1532 

3.0527 

2.9477 

2.8932 

2.8373 

2 . 7797 

2.7204 

2.6590 

2.5955 

14 

3.1469 

3.0501 

2.9493 

2.8437 

2.7888 

2.7324 

2.6742 

2.6142 

2.5519 

2.4872 

15 

3.0602 

2.9633 

2.8621 

2.7559 

2.7006 

2 . 6437 

2.5850 

2.5242 

2.4611 

2.3953 

16 

2.9862 

2.8890 

2.7875 

2.6808 

2.6252 

2.5678 

2.5085 

2.4471 

2.3831 

2.3167 

17 

2.9222 

2.8249 

2.7230 

2.6158 

2.5598 

2.5021 

2.4422 

2.3801 

2.3153 

2.2474 

18 

2.8664 

2.7689 

2.6667 

2.5590 

2.5027 

2.4445 

2 . 3842 

2.3214 

2.2558 

2.1869 

19 

2.8173 

2.7196 

2.6171 

2.5089 

2.4523 

2 . 3937 

2 . 3329 

2.2695 

2.2032 

2.1333 

20 

2.7737 

2.6758 

2.5731 

2 . 4645 

2.4076 

2 . 3486 

2.2873 

2.2234 

2.1562 

2  1 

2.7348 

2.6368 

2.5338 

2.4247 

2.3675 

2.3082 

2.2465 

2.1819 

2.1141 

22 

2.6998 

2.6017 

2.4984 

2.3890 

2.3315 

2.2718 

2.2097 

2.1446 

2.0760 

23 

2.6682 

2.5699 

2.4665 

2.3567 

2.2989 

2.2389 

2.1763 

2.1107 

2.0415 

1.9677 

24 

2.6396 

2.5412 

2.4374 

2.3273 

2.2693 

2.2090 

2.1460 

2.0799 

2,OOgg 

1.9353 

25 

2.6135 

2.5149 

2.4110 

2.3005 

2.2422 

2.1816 

2.1183 

2 .0517 

1.9811 

1.9055 

26 

2.5895 

2.4909 

2.3867 

2.2759 

2.2174 

2.1565 

2.0928 

2.0257 

1.9545 

1.8781 

27 

2.5676 

2.4688 

2.3644 

2.2533 

2.1946 

2.1334 

2.0693 

2.0018 

1.9299 

1.8527 

28 

2 . 5473 

2.4484 

2.3438 

2.2324 

2.1735 

2.1  121 

2.0477 

1.9796 

1.9072 

1.8291 

29 

2.5286 

2.4295 

2.3248 

2.2131 

2.1540 

2.0923 

2.0276 

1.9591 

1.8861 

30 

2.5112 

2.4120 

2.3072 

2.1952 

2. 1359 

2.0739 

2.0089 

1.9400 

1,8664 

1.7867 

40 

2.3882 

2.2882 

2.1819 

2.0677 

2.0069 

1.9429 

1.8752 

1.8028 

1.7242 

1.6371 

60 

2.2702 

2.1692 

2.0613 

1.9445 

1.8817 

1.8152 

1.7440 

1.6668 

1.5810 

1.4822 

120 

2.1570 

2.0548 

1.9450 

1.8249 

1.7597 

l . 6899 

1.6141 

l . 5299 

1.4327 

1.3104 

0O 

2.0483 

1.9447 

1.8326 

1.7085 

1.6402 

1.5660 

1.4835 

1.3883 

1.2684 

1.0000 
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TABLE  C-4  (continued) 


F  Distribution:  Upper  5  Percent  Points 


Degrees  of  Freedom  in  Numerator,  v 


1 

2 

3 

4 

5 

6 

7 

8 

9 

1 

161.45 

199.50 

215.71 

224.58 

230.16 

233.99 

236.77 

238.88 

240.54 

2 

18.513 

19.000 

19.164 

19.247 

19.296 

19.330 

19.353 

19.371 

19.385 

3 

10.128 

9.5521 

9.2766 

9.1172 

9-0135 

8.9406 

8.8868 

8.8452 

8.8123 

4 

7.7086 

6  .9443 

6.5914 

6.3883 

6.2560 

6.1631 

6.0942 

6.0410 

5.9988 

5 

6.6079 

5.7861 

5.4095 

5.1922 

5.0503 

4.9503 

4.8759 

4.8183 

4.7725 

6 

5.9874 

5-1433 

4.7571 

4.5337 

4.3874 

4.2839 

4.2066 

4.1463 

4.  0990 

7 

5-5914 

4.7374 

4 .3468 

4.1203 

3.9715 

3.8660 

3.7870 

3.7257 

3-6767 

8 

5-3177 

4.4590 

4.0662 

3.8378 

3.6875 

3.5806 

3. 5005 

3.4381 

3.3881 

9 

5.1174 

4.2565 

3.8626 

3.6331 

3.4817 

3.3738 

3.2927 

3.2296 

3-1789 

10 

4.9646 

4.1028 

3.7083 

3.4780 

3.3258 

3.2172 

3.1355 

3. 0717 

3.0204 

11 

4.8443 

3. 9823 

3- 5874 

3-3507 

3.2039 

3.0946 

3.0123 

2.9480 

2.8962 

12 

4.7472 

3.8853 

3-4903 

3.2592 

3.1059 

2.9961 

2.9134 

2.8486 

2.7964 

13 

4.6672 

3.8056 

3.4105 

3.1791 

3.0254 

2.9153 

2.8321 

2.7669 

2.7144 

14 

4.6001 

3.7389 

3.3439 

3.1122 

2.9582 

2.8477 

2.7642 

2.6987 

2.6458 

15 

4.5431 

3.6823 

3.2874 

3.0556 

2.9013 

2.7905 

2.7066 

2.6408 

2.5876 

16 

4.4940 

3.6337 

3.2389 

3  .0069 

2.8524 

2.7413 

2.6572 

2.5911 

2.5377 

17 

4.4513 

3-5915 

3.1968 

2.9647 

2.8100 

2  .6987 

2.6143 

2.5480 

2.4943 

18 

4.4139 

3- 5546 

3- 1599- 

2.9277 

2.7729 

2.6613 

2.5767 

2.5102 

2.4563 

19 

4.3803 

3.5219 

3.1274 

2.8951 

2.7401 

2.6283 

2.5435 

2.4768 

2.4227 

20 

4.3513 

3.4928 

3.0984 

2.8661 

2.7109 

2.5990 

2.5140 

2.4471 

2.3928 

21 

4.3248 

3.4668 

3.0725 

2.8401 

2.6848 

2.5727 

2.4876 

2.4205 

2.3661 

22 

4.3009 

3,4434 

3.0491 

2.8187 

2.6613 

2  .5491 

2.4638 

2.3965 

2.3419 

23 

4-2793 

3.4221 

3.0280 

2.7955 

2.6400 

2.5277 

2.4422 

2.3748 

2.3201 

24 

4.2597 

3.4028 

3.0088 

2.7763 

2.6207 

2.5082 

2.4226 

2.3551 

2.3002 

25 

4.2417 

3-3852 

2  .9912 

2.7587 

2.6030 

2  .4904 

2.4047 

2.3371 

2.2821 

26 

4.2252 

3.3690 

2.9751 

2  .7426 

2.5868 

2  .4741 

2.3883 

2.3205 

2.2655 

27 

4.2100 

3-3541 

2.9604 

2.7278 

2.5719 

2.4591 

2.3732 

2.3053 

2.2501 

28 

4.1960 

3.3404 

2.9467 

2.7141 

2.5581 

2.4453 

2.3593 

2.2913 

2.2360 

29 

4.1830 

3-3277 

2.9340 

2.7014 

2.5454 

2.4324 

2.3463 

2.2782 

2.2229 

30 

4.1709 

3.3158 

2.9223 

2.6896 

2.5336 

2.4205 

2.3343 

2.2662 

2.2107 

40 

4.0848 

3.2317 

2.8387 

2.6060 

2.4495 

2.3359 

2.2490 

2.1802 

2.1240 

60 

4.0012 

3.1504 

2.7581 

2.5252 

2.3683 

2.2540 

2.1665 

2 . 0970 

2.0401 

L20 

3.9201 

3.0718 

2.6802 

2.4472 

2.2900 

2.1750 

2.0867 

2  .0164 

1.9588 

CO 

3.8415 

2.9957 

2.6049 

2.3719 

2.2141 

2.0986 

2.0096 

1.9384 

1.8799 
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TABLE  C-4  (continued) 

F  Distribution:  Upper  5  Percent  Points 


Degrees  of  Freedom  in  Numerator,  v . 


10 

12 

15 

20 

24 

30 

40 

60 

120 

OO 

1 

241.88 

245.95 

248.01 

249.05 

250.  og 

251.14 

252.20 

253.25 

254.32 

2 

19.396 

19.429 

19.446 

19.454 

19.462 

19.471 

19.479 

19.487 

19.496 

3 

8.7855 

8.7446 

8.7029 

8.6602 

8.6385 

8.6166 

8.5944 

8.5720 

8.5494 

8.5265 

4 

5.9644 

5.9117 

5.8578 

5.8025 

5-7744 

5-7459 

5.7170 

5.6878 

5.6581 

5.6281 

5 

4.7351 

4.6777 

4.6188 

4.5581 

4.5272 

4.4957 

4.4638 

4.4314 

4.3984 

4.3650 

6 

3-9999 

3.9381 

3.8742 

3.8415 

3.8082 

3-  7743 

3.7398 

3.7047 

3.6688 

7 

3.6365 

3.5747 

3.5108 

3.4445 

3.4105 

3-3758 

3.3404 

3. 3043 

3.2674 

3.2298 

8 

3.3472 

3.2840 

3.2184 

3.1503 

3.1152 

3.  0794 

3.0428 

3.0053 

2.9669 

2.9276 

9 

3.1373 

3.0729 

3.0061 

2.9365 

2.9005 

2.8637 

2.8259 

2.7872 

2.7473 

2.7067 

10 

2.9782 

2.9130 

2.8450 

2.7740 

2.7372 

2.6996 

2.6609 

2.6211 

2.5801 

2.5379 

'll 

2  .8536 

2  .7876 

2.7186 

2.6464 

2.6090 

2.5705 

2.5309 

2.4901 

2.4480 

2.4045 

12 

2.7534 

2.6866 

2.6169 

2.5436 

2-5055 

2.4663 

2.4259 

2  .3842 

2.3410 

2.2962 

13 

2.6710 

2  .6037 

2.5331 

2.4589 

2.4202 

2.3803 

2.3392 

2.2966 

2.2524 

2.2064 

14 

2.6021 

2.5342 

2.4630 

2.3879 

2.3487 

2.3082 

2.2664 

2.2230 

2.1778 

2.1307 

15 

2.5437 

2.4753 

2.4035 

2.3275 

2.2878 

2.2468 

2.2043 

2.1601 

2.1141 

2.0658 

16 

2.4935 

2  .4247 

2.3522 

2.2756 

2.2354 

2.1938 

2.1507 

2.1058 

2.0589 

2.0096 

17 

2.4499 

2  .3807 

2.3077 

2.2304 

2.1898 

2.1477 

2.1040 

2.0584 

2.0107 

1.9604 

18 

2.4117 

2.3421 

2.2686 

2.1906 

2.1497 

2.1071 

2.0629 

2.0166 

1.9681 

1.9168 

19 

2.3779 

2. 3080 

2.2341 

2.1555 

2.1141 

2.0712 

2 .0264 

1.9796 

1.9302 

1.8780 

20 

2.3479 

2  .2776 

2.2033 

2.1242 

2.0825 

2.0391 

1.9938 

1.9464 

1.8963 

1 .8432 

21 

2.3210 

2.2504 

2.1757 

2.0960 

2.0540 

2 .0102 

1.9645 

1.9165 

1.8657 

1.8117 

22 

2.2967 

2.2258 

2.1508 

2.0707 

2.0283 

1 .9842 

1.9380 

1 .8895 

1.8380 

1.7831 

23 

2  .2747 

2.2036 

2.1282 

2.0476 

2.0050 

1.9605 

1.9190 

1.8649 

1.8128 

1.7570 

24 

2.2547 

2.1834 

2.1077 

2.0267 

1 .9838 

1.9390 

1 .8920 

1.8424 

1. 7897 

1.7331 

25 

2.2365 

2.1649 

2.0889 

2.0075 

1.9643 

1.9192 

1.8718 

1.8217 

1 .7684 

1.7110 

26 

2.2197 

2.1479 

2.0716 

1.9898 

1.9464 

1.9010 

1.8533 

1.8027 

1.7488 

1 .6906 

27 

2.2043 

2.1323 

2.0558 

1.9736 

1.9299 

1.8842 

1.8361 

1.7851 

1.7307 

1.6717 

28 

2.1179 

2.0411 

1. 9586 

1.9147 

1 .8687 

1.8203 

1.7689 

1.7138 

1.6541 

29 

2.1768 

2  .1045 

2.0275 

1.9446 

1.9005 

1.8543 

1.8055 

1.7537 

1.6981 

1.6377 

30 

2.1646 

2.0921 

2.0148 

1.9317 

1.8874 

1.8409 

1.7918 

1.7396 

1.6335 

1.6223 

40 

2  0772 

2  .0035 

1.9245 

1.8389 

1.7929 

1.7444 

1.6928 

1.6373 

1.5766 

1.5089 

60 

1.9926 

1.9174 

1.8364 

1.7480 

1.7001 

1.6491 

1. 5943 

1.5343 

1.4673 

1-3893 

12d 

1.9105 

1.8337 

1.7505 

1.6587 

1.6084 

1.5543 

1.4952 

1.4290 

1.3519 

1. 2539 

00 

1.8307 

1.7522 

1.6664 

1.5705 

1.5173 

1.4591 

1.3940 

1.3180 

1.2214 

1.0000 
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TABLE  C-4  (continued) 


F  Distribution:  Upper  10  Percent  Points 


Degrees  of  Freedom  in  Numerator  1 


1 

2 

3 

4 

5 

6 

7 

8 

9 

1 

39.864 

49.500 

53.593 

55.833 

57.241 

58.204 

58.906 

59-439 

59.858 

2 

8.5263 

9.0000 

9.1618 

9.2434 

9.2926 

9.3255 

9.3491 

9.3668 

9.3805 

3 

5-5383 

5.4624 

5. 3908 

5.3427 

5.3092 

5.2847 

5.2662 

5.2517 

5.2400 

4 

4.5448 

4.3246 

4.1908 

4.1073 

4.0506 

4.0098 

3-9790 

3.9549 

3.9357 

5 

4.0604 

3-7797 

3.6195 

3.5202 

3.4530 

3.4045 

3.3679 

3.3393 

3.3163 

6 

3-7760 

3.4633 

3.2888 

3  .1808 

3.1075 

3.0546 

3.0145 

2.9830 

2.9577 

7 

3.5894 

3.2574 

3.0741 

2.9605 

2.8833 

2.8274 

2.7849 

2.7516 

2.7247 

8 

3-4579 

3.1131 

2.9238 

2.8064 

2.7265 

2.6683 

2.6241 

2.5893 

2.5612 

9 

3-3603 

3.0065 

2.8129 

2.6927 

2.6106 

2.5509 

2.5053 

2.4694 

2.4403 

10 

3.2850 

2.9245 

2.7277 

2.6053 

2,5216 

2.4606 

2,4140 

2.3772 

2  e3473 

11 

3.2252 

2.8595 

2.6602 

2.5362 

2.4512 

2.3891 

2.3416 

2.3040 

2.2735 

12 

3-1765 

2.8068 

2.6055 

2.4801 

2.3940 

2.3310 

2.2828 

2.2446 

2.2135 

13 

3.1362 

2.7632 

2.5603 

2.4337 

2.3467 

2.2830 

2.2341 

2.1953 

2 . 1638 

14 

3.1022 

2.7265 

2.5222 

2.3947 

2.3069 

2.2426 

2.1931 

2.1539 

2.1220 

15 

3.0732 

2.6952 

2.4898 

2.3614 

2.2730 

2.2081 

2.1582 

2.1185 

2,0862 

16 

3.0481 

2  .6682 

2.4618 

2.3327 

2.2438 

2.1783 

2.1280 

2.0880 

2.0553 

17 

3.0262 

2,6446 

2.4374 

2.3077 

2.2183 

2.1524 

2.1017 

2.0613 

2.0284 

18 

3,0070 

2.6239 

2,4160 

2.2858 

2  ,1958 

2.1296 

2.0785 

2.0379 

2.0047 

19 

2.9899 

2.6056 

2.3970 

2.2663 

2.1760 

2.1094 

2.0580 

2.0171 

1 .9836 

20 

2  .9747 

2.5893 

2.3801 

2.2489 

2.1582 

2.0913 

2.0397 

1.9985 

1.9649 

21 

2.9609 

2  .5746 

2.3649 

2.2333 

2.1423 

2.0751 

2.0232 

1.9819 

1.9480 

22 

2.9486 

2.5613 

2.3512 

2.2193 

2.1279 

2.0605 

2  .0084 

1 .9668 

1.9327 

23' 

2-9374 

2.5493 

2.3387 

2.2065 

2.1149 

2.0472 

1.9949 

1.9531 

1.9189 

24 

2.9271 

2.5383 

2.3274 

2.1949 

2.1030 

2.0351 

1 ,9826 

1 .9407 

1.9063 

25 

2.9177 

2  .5283 

2.3170 

2,1843 

2.0922 

2.0241 

1.9714 

1 .9292 

1 .8947 

26 

2.9091 

2.5191 

2.3075 

2.1745 

2.0822 

2.0139 

1.9610 

1.9188 

1,8841 

27 

2.9012 

2.5106 

2.2987 

2  .1655 

2.0730 

2.0045 

1.9515 

1 .9091 

1.8743 

28 

2.8939 

2.5028 

2.2906 

2.1571 

2.0045 

1.9959 

1.9427 

1.9001 

1 .8652 

29 

2.8871 

2.4955 

2.2831 

2.1494 

2  .0566 

1.9878 

1.9345 

1.8918 

1 .8568 

30 

2.8807 

2.4887 

2.2761 

2.1422 

2.0492 

1.9803 

1 .9269 

1.8841 

1.8490 

40 

2.8354 

2.4404 

2.2261 

2.0909 

1.9968 

1.9269 

1 .8725 

1.8289 

1.7929 

60 

2.7914 

2.3932 

2.1774 

2.0410 

1.9457 

1.8747 

1.8194 

1.7748 

1.7380 

120 

2.7478 

2.3473 

2.1300 

1.9923 

1 .8959 

1 .8238 

1.7675 

1 .7220 

1 .6843 

03 

2.7055 

2,3026 

2.0838 

1.9449 

1.8473 

1.7741 

1.7167 

1 .6702 

1.6315 
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TABLE  C-4  (continued) 

F  Distribution:  Upper  10  Percent  Points 


Degrees  of  Freedom  in  Numerator  v  ^ 


10 

12 

15 

20 

24 

30 

40 

60 

120 

m 

1 

60.195 

60.705 

61.220 

61.740 

62.002 

62.265 

62,529 

62.794 

63.061 

63 . 328 

2 

9,3916 

9 .4081 

9.4247 

9.4413 

9.4496 

9.4579 

9.4663 

9 .4746 

9.4829 

9.  4913 

3 

5.2304 

5.2156 

5.2003 

5 .1845 

5  .1764 

5.1681 

5.1597 

5.1512 

5  .1425 

5. 1337 

4 

3.9199 

3.8955 

3.8689 

3.8443 

3.8310 

3.8174 

3.8036 

3.7896 

3.7753 

3.7607 

5 

3.2974 

3.2682 

3.2380 

3.2067 

3.1905 

3.1741 

3. 1573 

3,1402 

3.1228 

3.1050 

6 

2 .9369 

2,9047 

2.8712 

2.8363 

2.8183 

2.8000 

2.7812 

2.7620 

2.7423 

2.7222 

7 

2.7025 

2.6681 

2.6322 

2  e5947 

2.5753 

2.5555 

2.5351 

2.5142 

2.4928 

2.4708 

8 

2.5380 

2.5020 

2.4642 

2.4246 

2.4041 

2.3830 

2.3614 

2.3391 

2.3162 

2.2926 

9 

2.4163 

2.3789 

2.3396 

2.2983 

2.2768 

2.2547 

2.2320 

2.2085 

2.1843 

2.1592 

10 

2.3226 

2.2841 

2.2435 

2.2007 

2 . 1784 

2.1554 

2.1317 

2.1072 

2.0818 

2,0554 

11 

2.2482 

2.2087 

2.1671 

2.1230 

2.1000 

2.0762 

2.0516 

2.0261 

1.9997 

1.9721 

12 

2.1878 

2.1474 

2.1049 

2.0597 

2.0360 

2.0115 

1.9861 

1.9597 

1.9323 

1.9036 

13 

2.1376 

2.0966 

2.0532 

2.0070 

1.9827 

1.9576 

1 . 9315 

1.9043 

1,8759 

1.8462 

14 

2.0954 

2  .0537 

2  .0095 

1.9625 

1  •  9377 

1.9119 

1.8852 

1.8572 

1,8280 

1.7973 

15 

2.0593 

2 .0171 

1.9722 

1.9243 

1.8990 

1.8728 

1.8454 

1.8168 

1.7867 

1.7551 

16 

2.0281 

1.9854 

1.9399 

1.8913 

1 .8656 

1.8388 

1.8108 

1.7816 

1.7507 

1.7’ 

17 

2,0009 

1.9577 

1.9117 

1.8624 

1.8362 

1.8090 

1.7805 

1.7506 

1.7191 

1.6t 

18 

1.9770 

1  .9333 

1.8868 

1.8368 

1.8103 

1.7827 

1  e7537 

1.7232 

1.6910 

1.6567 

19 

1.9557 

1.9U7 

1.8647 

1.8142 

1.7873 

1 . 7592 

1.7298 

1.6988 

1.6659 

1.6308 

20 

1.9367 

1.8924 

1.8449 

1  .7938 

l  .7667 

1.7382 

1.7083 

1,6768 

1.6433 

1.6074 

21 

1  •  9197 

1.8750 

1.8272 

1.7756 

1.7481 

1.7193 

1 ,6890 

1.6569 

1.6228 

1.5862 

22 

1.9043 

1.8593 

1.8111 

1.7590 

1.7312 

1.7021 

1.6714 

1.6389 

1,  6042 

1.5668 

23 

1.8903 

1.8450 

1 ,7964 

1.7439 

1.7159 

1.6864 

1.6554 

1,6224 

1.5871 

1.5490 

24 

1.8775 

1.8319 

1,7831 

1.7302 

1.7019 

1.6721 

1.6407 

1.6073 

1.5715 

1 . 5327 

25 

1.8658 

1.8200 

1.7708 

1.7175 

1.6890 

1,6589 

1.6272 

1  .5934 

1.5570 

1.5176 

26 

1.8550 

1.8090 

1.7596 

1.7059 

1  .6771 

1.6468 

1.6147 

1  .5805 

1 .5437 

1.5036 

27 

1.8451 

1.7989 

1.7492 

1.6951 

1.6662 

1.6356 

1.6032 

1  .5686 

1.5313 

1.4906 

28 

1.8359 

1.7895 

1.7395 

1.6852 

1.6560 

1.6252 

1.5925 

1.5575 

1.5198 

1.4784 

29 

1.8274 

1.7808 

1.7306 

1.6759 

1.6465 

1  .6155 

1 .5825 

1.5472 

1.5090 

1.4670 

30 

1.8195 

1.7727 

1.7223 

1 .6673 

1.6377 

1.6065 

1 . 5732 

1.5376 

1 .4989 

1.4564 

40 

1.7627 

1.7146 

1  .6624 

1 .6052 

1.5741 

1.5411 

1.5056 

1.4672 

1,4248 

1.3769 

60 

1.7070 

1.6574 

1.6034 

1.5435 

1.5107 

1.4755 

1.4373 

1.3952 

1 .3476 

1.2915 

120 

1.6524 

1.6012 

1.5450 

1.4821 

1.4472 

1.4094 

1.3676 

1.3203 

1.2646 

1.1926 

00 

1 .5987 

1.5458 

1.4871 

1.4206 

1.3832 

1.3419 

1.2951 

1.2400 

1.1686 

1.0000 
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TABLE  C-5 

CRITICAL  VALUES  da(n)  CF  THE  MAXIMUM  ABSOLUTE 
DIFFERENCE  BETWEEN  SAMPLE  AND  POPULATION 
RELIABILITY  FUNCTIONS 

Sample 

Size 

(N) 

Level 

of  Significance  (a) 

0.20 

0.15 

0.10 

0.05 

0.01 

3 

O.565 

0.597 

0.642 

0.708 

0.828 

4 

0  .494 

0.525 

0.564 

0  .624 

0.733 

5 

0  .446 

0.474 

0  .474 

0.565 

0 .669 

10 

0.322 

0.342 

0.368 

0.410 

0.490 

15 

0.266 

0.283 

0.304 

0.338 

0.404 

20 

0.231 

0.246 

0.264 

0.294 

0.356 

25 

0.21 

0.22 

0.24 

0.27 

0.32 

30 

0.19 

0.20 

0.22 

0.24 

0.29 

35 

0.18 

0.19 

0.21 

0.23 

0.27 

40 

0.17 

0.18 

0.19 

0.21 

0.25 

45 

0.16 

0.17 

0.18 

0.20 

0.24 

50 

0.15 

0.16 

0.17 

0.19 

0.23 

Over") 

1.07 

1.14 

1.22 

1.36 

1.63 

50  J 

Vn 

■/n 

/n 

yn 

yn 
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TABLE  C-6 


Reliability  Exponential  Function  (R  =  e-Xt  j 


At 

R 

At 

R 

O.OOI 

0  .gggooo 

0.051 

0.950279 

0.002 

0  .998002 

0.052 

0.949329 

0.003 

0.997004 

0.053 

0.948380 

0.004 

0.996008 

omk 

O.947432 

0.005 

0.995012 

0.055 

0.946485 

0.006 

0. 994018 

0.056 

0.94  5539 

0  .007 

0.993024 

0.057 

0.94  4594 

0.008 

0.992032 

0.058 

0.943650 

0.009 

0.991040 

0.059 

0 .942709 

0.010 

0.990050 

0.060 

0.941765 

0.011 

0.989060 

0.061 

0.940823 

0.912 

0.988072 

0.062 

0.939883 

0  .013 

0.987084 

0.063 

0.938943 

0.014 

0.986098 

0.064 

0  .938005 

0.015 

0.985112 

O.O65 

0.937067 

0.016 

0.984127 

0.066 

0.936131 

0.017 

0.983144 

0.067 

0.935195 

0.018 

0.982161 

0.068 

0.934260 

0.019 

0.981179 

0.069 

0.933329 

0.020 

0.980199 

0.070 

0.932394 

0.021 

0.979219 

0.071 

0.931462 

0.022 

0.978240 

0.072 

0.930531 

0.023 

0.977262 

0.073 

0.929601 

0.024 

0.976286 

0.074 

0.928677 

0.025 

0.975310 

0.075 

0.927743 

0.026 

0.974335 

0.076 

0.9268  16 

0.027 

0.973361 

0.077 

0.925890 

0.028 

0.972388 

0.078 

0.924964 

0.029 

0.971416 

0.079 

0.924040 

0.030 

0.970446 

0.080 

0.923116 

0.031 

0.969476 

0.081 

0.922194 

0.032 

0.968507 

0.082 

0.921272 

0.033 

0.967539 

0.083 

0.920351 

0.034 

0.966572 

0.084 

0.919431 

0.035 

0.965605 

0.085 

0.918512 

0.036 

0.964640 

0.086 

0.917594 

0.037 

0.963676 

0.087 

0.916677 

0.038 

0.962713 

0.088 

0.915761 

0.039 

0.961751 

0.089 

0.914846 

0.040 

0  .960789 

0.090 

0.913931 

0.041 

0.959829 

0.091 

0.913018 

0.042 

0.958870 

0.092 

0.912105 

0.043 

0.9579H 

0. 093 

0.911194 

0.044 

0 . 956954 

0.094 

0.910283 

0.045 

0.955997 

O.095 

0.909373 

0.046 

0.955942 

0.096 

0.908464 

0.047 

0.954087 

0.097 

0.907556 

0.048 

0.953134 

0.098 

0.906649 

0.049 

0.952181 

O.O99 

0.905743 

0.050 

0.951229 

0.100 

0.904837 
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TABLE  C-6  (continued) 


Reliability  Exponential  Function  (R  =  e  (Cont.  ) 


xt 

R 

Xt 

R 

0.110 

0.895834 

0.65 

0.522046 

0.120 

0.886920 

0.70 

0.496585 

0.130 

0.878095 

0.75 

0.472367 

0.140 

0.869358 

0.80 

0. 449329 

0.150 

0.860708 

0.85 

0. 427415 

0.160 

0.852144 

0.90 

0.406570 

0.170 

0.843665 

0.95 

0.386741 

0.180 

0.835270 

1.00 

0.367878 

0.190 

0.826959 

1.05 

0.34993 

0.200 

0.818731 

1.10 

0.332871 

0.210 

0.810584 

1.15 

0.316637 

0.220 

0.802519 

1.20 

0.301194 

0.230 

0.794534 

1.25 

0.286505 

0.240 

0.786628 

1.30 

0.272532 

0.250 

0.778801 

1.35 

0.259240 

0.260 

0 .771052 

1.40 

0.246597 

0.270 

0.763379 

1.45 

0.234570 

0.280 

0.755784 

1.50 

0.223130 

0.290 

0.748264 

1.55 

0.212248 

0.300 

0.740818 

1.60 

0.201897 

0.310 

0.733447 

1.70 

0.182684 

0.320 

0.725149 

1.80 

0. 165299 

0.330 

0.718924 

1.90 

0. 149569 

0.340 

O.7II77O 

2.00 

0.350 

0.704688 

2.10 

0. 122456 

0.360 

0.697676 

2.20 

0.110803 

0.370 

0.690734 

2.30 

0.100259 

0.380 

0.683861 

2.40 

0.090718 

0.390 

0.677057 

2.50 

0.082085 

0.400 

0.670320 

2.60 

0.074274 

0.410 

0 .663650 

2.70 

0.067206 

0.420 

0. 657047 

2.80 

0.060810 

0.430 

2.90 

0.055023 

0  .440 

0.644036 

3.00 

0.049787 

0.450 

0.637628 

3.25 

0.038774 

0.460 

0.631284 

3.50 

0 . 030197 

0.470 

0. 625002 

3.75 

0.023518 

0.480 

0.618783 

4.00 

0.018316 

0.490 

0.612626 

4.25 

0.014264 

0.500 

0.606531 

4.50 

0. 011109 

0.510 

0.600496 

4.75 

0.008652 

0.520 

0.594521 

5.00 

0.006738 

0.530 

0.588605 

5.50 

0.004087 

0.540 

0.582748 

6.00 

0.002479 

0.550 

0.576950 

6.50 

0.001503 

0.560 

O'.  571209 

7.00 

0.000912 

0.570 

0. 565525 

7.50 

0.000553 

0.580 

0.559898 

8.00 

0.000335 

0.590 

0.554327 

9.00 

0.000123 

0.600 

0.548812 

10.0 

0.000045 
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TABLE  C-7 

FACTORS  FOR  CALCULATION  OF  MEAN  LIFE 
CONFIDENCE  INTERVALS  FROM  TEST  DATA  [Factors  = 

(Exponential  Distribution  Assumed) 


2/X2(p  ,d)] 


a. 

c* 

t-, 

95^  Two-sided 

91-1/2% 

One-sided 

o 

90%  Two-Sided 

95%  One 

-sided 

w 

o 

80%  Two-sided 

90^  One 

-sided 

u 

to 

0) 

Q 

60% Two-sided 
80% One-sided 

Lower 

Limit 

Upper  Limit 

2 

.185 

.217 

.272 

•333 

.433 

.619 

4.47 

9.462 

19.388 

59.58 

100.0 

200.0 

4 

.135 

.151 

.180 

.210 

.257 

.334 

1.21 

1.882 

2.826 

4.102 

6.667 

10.00 

6 

.108 

.119 

.139 

.159 

.188 

.234 

.652 

.909 

1.221 

1.61 3 

2.3  077 

3.007 

8 

.0909 

.100 

.114 

.129 

.150 

.181 

.437 

.573 

0.733 

.921 

1.212 

1.481 

10 

,0800 

.0857 

.0976 

.109 

.125 

.149 

.324 

.411 

.508 

.600 

.789 

.909 

12 

.0702 

.0759 

.0856 

.0952 

JC7 

.126 

.256 

.317 

•  383 

.454 

.555 

.645 

14 

.0635 

.0690 

.0765 

.0843 

.0948 

.109 

.211 

.257 

.305 

•  355 

■  431 

.500 

16 

.0588 

.0625 

.0693 

.0760 

.0848 

.0976 

.179 

.215 

.251 

.290 

.345 

.385 

18 

,0536 

.0571 

•  0633 

.0693 

.0769 

.0878 

.156 

.184 

.213 

.243 

.286 

•322 

20 

.0500 

.05 3  7 

.0585 

.0635 

.0703 

.0799 

.137 

.158 

.184 

.208 

.242 

.270 

22 

.0465 

.0495 

.0543 

.0589 

.0648 

.0732 

.123 

.142 

.162 

.182 

.208 

.232 

24 

.0439 

.0463 

.0507 

.0548 

.0601 

.0676 

.111 

.128 

.144 

.161 

.185 

.200 

26 

.0417 

.0438 

.0476 

.0513 

.0561 

.0629 

.101 

.116 

.130 

.144 

.164 

.178 

28 

.0392 

.0413 

.0449 

.0483 

.0527 

.0588 

.0927 

.106 

.118 

■  131 

.147 

.161 

30 

.03  75 

.0393 

.0425 

.0456 

.0496 

.0551 

.0856 

.0971 

.108 

.119 

.133 

.145 

32 

.03  55 

.0374 

.0404 

.0433 

.0469 

.0519 

.0795 

.0899 

.0997 

.109 

.122 

.131 

34 

.033  9 

.03  57 

.03  85 

.0411 

.0445 

.0491 

.0742 

.0834 

.0925 

.101 

.1 13 

.122 

36 

.03  2  5 

.0342 

.03  67 

.03  92 

.0423 

.0466 

.0696 

.0781 

.0899 

.0939 

.104 

.111 

38 

.03  11 

.03  2  7 

.0351 

.0375 

.0404 

.0443 

.0656 

.0732 

.0804 

.0874 

.0971 

.103 

;o 

.0299 

.0314 

.0337 

.0359 

.0 386 

.0423 

.0619 

.0689 

.0756 

.0820 

.0901 

.0968 

To  Use:  Multiply  value  shown  by  total  part  hours  to  get  MIBF  figures  in  hours 

Note:  d=2r,  except  for  the  lower  limit  on  tests  truncated  at  a  fixed  time  and  where  r<n.  In  such  cases,  use 
d=2(r+l). 


AMCP  706-191 


AMCP  706-191 


TABLE  C-8 

REQUIRED  NUMBER  CF  FAILURES  FOR  VARIOUS  VALUES 

CF  CONFIDENCE  AND  PRECISION 

(Exponential  Distribution) 

Precision-  6 

Confidence 

85% 

90% 

95  $ 

99% 

5% 

830 

1082 

1537 

2655 

10$ 

207 

271 

384 

664 

15% 

92 

120 

171 

295 

20% 

52 

67 

96 

166 

25% 

33 

43 

61 

106 

30% 

23 

30 

43 

74 

35% 

17 

22 

31 

54 

Example:  43  failures  are  required 

confident  that  the  estimated  MIBF 

the  true  value. 

to  be  90$ 
i  s  within  25$  of 
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Values  of  r(n 

TABLE  0-9 

GAMMA  FUNCTION 

)  =  r°°e-x  x11'1  dx; 
■>  0 

r(n  +  1)  = 

nT(n) 

n 

T(n) 

n 

r(n) 

n 

r(n) 

n 

r(n) 

1.00 

1.00000 

1.25 

0.90640 

1.50 

0.88623 

1.75 

0.91906 

1.01 

0.99433 

1.26 

0.90440 

1.51 

0.88659 

1.76 

0.92137 

1.02 

0.98884 

1.27 

0.90250 

1.52 

0.88704 

1.77 

0. 92376 

1.03 

0.98355 

1.28 

0.90072 

1.53 

0.88757 

1.78 

0.92623 

1.04 

0.97844 

1.29 

0.89904 

1.54 

0.88818 

1.79 

0.92877 

1.05 

0.97350 

1.30 

0.89747 

1.55 

0.88887 

1.80 

0.93138 

1.06 

0.96874 

1.31 

0.80600 

1.56 

0.88964 

1.81 

0.93408 

1.07 

0.96516 

1.32 

0.89464 

1.57 

0.89049 

1.82 

0.93685 

1.08 

0 . 95973 

1  a33 

0.89338 

1.58 

0.89142 

1.83 

0.93969 

1.09 

0.95546 

1.34 

0.89222 

1.59 

0.89243 

1.84 

0.94261 

1.10 

0.95135 

1.35 

0.89115 

1.60 

0.89352 

1.85 

0.94561 

1.11 

0.94739 

1.36 

0.89018 

1.61 

0.89468 

1.86 

0.94869 

1.12 

0 . 94359 

1.37 

0.88031 

1.62 

0.89592 

1.87 

0.95184 

1.13 

0.93993 

1.38 

0.88854 

1.63 

0.89724 

1.88 

0.95507 

1.14 

0.93642 

1.39 

0.88785 

1.64 

0.89864 

1.89 

0.95838 

1.15 

0 .93304 

1.40 

0.88726 

1.65 

0.90012 

1.90 

0.96177 

1.16 

0.92980 

1.41 

O.88676 

1.66 

0.90167 

1.91 

0.96523 

1.17 

0.92670 

1.42 

0.88636 

1.67 

0.90330 

1.92 

0.96878 

1.18 

0.92373 

1.43 

0.88604 

1.68 

0.90500 

1.93 

0.97240 

1.19 

0.92088 

1.44 

0.88580 

1.69 

0.90678 

1.94 

0.97610 

1.20 

0.91817 

1.45 

0.88565 

1.70 

0,90864 

1.95 

0.97988 

1.21 

0.91558 

1.46 

0.88560 

1.71 

0.91057 

1.96 

0  .98374 

1.22 

0.91311 

1.47 

0.88563 

1.72 

0.91258 

1.97 

0.98768 

1.23 

0.91075 

1.48 

0  .88575 

1.73 

0.91466 

1.98 

0,99171 

1.24 

0.90852 

1.49 

0.88595 

1.74 

0.91683 

1.99 

2.00 

0,99581 

1.00000 
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FIGURE  C-1A 

CONFIDENCE  BELTS  FOR  PROPORTIONS 

Confidence  Coefficients:  0.80 
for  Two-sided  Estimation,  0.90 
for  One-Sided  Estimation 
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Point  Estimate,  P 


FIGURE  C-1B 

CONFIDENCE  BELTS  FOR  PROPORTIONS 

Confidence  Coefficients:  0.90 
for  Two-sided  Estimation,  0.95 
for  One-sided  Estimation 
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Point  Estimate,  P 


FIGURE  C-1C 


CONFIDENCE  BELTS  FOR  PROPORTIONS 

Confidence  Coefficients:  0.95 
for  Two-sided  Estimation,  0.975 
for  One-Sided  Estimation 
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GUIDE  FOR  REVIEWERS  OF  STUDIES  CONTAINING 
COST-EFFECTIVENESS  ANALYSIS  * 

CHAPTER  I 

GENERAL  BACKGROUND 


Introduction 


To  assist  m  the  review  of  studies  containing  cost-effectiveness  analyses, 
a  series  of  key  questions  with  explanatory  notes  have  been  prepared  and  are 
contained  in  the  next  chapter.  These  questions,  taken  together,  will  not 
necessarily  cover  all  aspects  of  all  cost-'effectiveness  analyses.  No  one 
general  list  of  questions  can  do  that.  Rather,  the  questions  are  designed 
to  focus  the  attention  of  the  reviewer  on  selected  aspects  to  assist  him  in 
evaluating  the  analysis.  All  the  questions  are  not' applicable  to  all  studies 
and  they  are  not  necessarily  of  equal  importance  to  those  studies  where  they 
do  apply.  The  reviewer  must  exercise  his  judgment  on  whether  the  questions 
are  applicable  and  the  degree  of  applicability  to  the  study  being  reviewed. 
Those  questions  that  are  considered  particularly  important  and  of  widest 
application  have  been  underlined,  and  are  also  listed  separately  for  conven¬ 
ience  in  the  back  as  "SELECTED  QUESTIONS.  "**  This  document  is  intended 
only  as  a  guide  and  not  as  a  full  and  comprehensive  treatment  of  all  aspects 
of  cost-effectiveness  analysis. 

Questions  that  do  not  bear  on  military  cost-effectiveness  analyses  are 
not  included  in  the  next  chapter.  Furthermore,  no  questions  are  addressed 
to  the  subject  of  the  intuitive  judgment  and  other  factors  used  in  making 
decisions  to  which  cost-effectiveness  analyses  contribute. 

Cost-Effectiveness  Analysis  and  the  Estimate  of  the  Situation 

Cost-effectiveness  analysis  is  a  method  for  studying  how  to  make  the 
best  of  several  choices.  By  cost-effectiveness  is  meant  the  relation  of  the 
resources  required  (cost)  to  achieve  a  certain  ability  to  accomplish  an 
objective  (effectiveness).  The  term  cost-effectiveness  is  always  used  in 
relation  to  the  effectiveness  of  alternative  systems;  organizations,  or 
activities. 

Cost-effectiveness  analysis  is  based  on  the  economic  concept  that  all 
military  decisions  involve  the  allocation  (best  use)  of  limited  resources 
among  competing  requirements.  The  allocation  is  determined  by  studying 
howto  get  the  best  use  of  the  available  resources.  This  same  concept  is 
embodied  in  Army  decision  processes.  It  is  used  by  a  combat  commander 
when  he  determines  (estimate  of  the  situation)  the  allocation  of  his  resources 


xx  Questions  1,  4,  12,  13,  14,  22,  23,  31,  37,  44,  46,  57,  60,  66,  and  73. 

*Reproduced  by  permission  of  the  Research  Analysis  Corporation,  McLean,  va.. 
Authored  by  I.  Heymont,  0.  Bryk.  H.  Linstone,  and  J.  Surmeier. 
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(forces)  among  the  main  and  secondary  efforts  and  reserves  in  the  offense 
or  between  the  forward  and  reserve  forces  in  the  defense.  A  G3  uses  the 
same  concept  inpreparing  his  recommendations  for  reallocating  among  the 
elements  of  the  command  the  ammunition  available  supply  rate  announced 
by  the  higher  headquarters.  The  company  commander  goes  through  the  same 
process  in  deciding  how  to  spend  his  company  funds. 

Although  cost-effectiveness  analysis  and  the  estimate  of  the  situation 
are  similar  in  concept,  they  differ  in  several  aspects.  The  purpose  of 
an  estimate  of  the  situation  is  to  arrive  at  a  recommended  course  of  action. 

It  is  usually  a  process  to  arrive  at  decisions  to  solve  "today's  problems 
today."  It  does  not  concern  itself,  in  a  realistic  sense,  with  problems, 
operations,  or  systems  of  the  future,  even  though  it  is  sometimes  not 
clear  where  the  problem  of  today  ends  and  the  problem  of  tomorrow  starts. 
Because  it  deals  with  relatively  immediate  problems,  the  formulation  of 
possible  courses  of  action  in  an  estimate  of  the  situation  is  severely  limited. 
The  resources  (forces  and  weapons)  available  to  the  commander  are  fixed 
by  what  has  been  made  available  and  there  is  no  real  flexibility  in  changing 
their  composition  or  basic  organization.  In  practice,  it  is  also  usually 
difficult  to  obtain  additional  resources  from  the  next  higher  commander. 

Another  severe  constraint  on  the  estimate  of  the  situation  is  the  time 
factor.  Information  is  usually  incomplete  and  the  time  available  before  a 
decision  is  required  often  does  not  permit  filling  in  gaps — even  if  it  were 
possible.  Often  there  is  only  sufficient  time  to  analyze  the  mission,  gather 
staff  estimates,  formulate  a  few  possible  courses  of  action  and  quickly 
weigh  these  courses  of  action  against  the  enemy  capabilities  (or  difficulties 
to  be  overcome)  and  with  each  other,  and  select  a  course  of  action  based  on 
some  criteria--often  called  the  governing  factors.  Time  usually  does  not 
permit  testing  the  range  of  the  dependence  of  the  proposed  course  of  action 
on  the  staff  estimates  and  planning  assumptions. 

Military  cost-effectiveness  analysis  is  not  a  decision  process  but  an 
aid  in  facilitating  decisions  that  must  be  made  now  in  regard  to  development, 
force  composition,  and  logistical  and  manpower  policy  problems  in  order 
to  be  prepared  for  wars  in  the  future.  The  analytical  techniques  employed 
in  cost-effectiveness  analysis  are  required  to  supplement  those  employed 
in  the  estimate  of  the  situation  because,  as  we  look  into  the  future,  the 
number  of  uncertainties  multiply.  These  uncertainties  include  such  things 
as  planning  factors,  the  enemy  and  his  reactions,  the  strategic  concept, 
technology,  change,  and  even  the  national  objectives  which  can  be  expected 
to  change  in  the  future  as  alliances  shift  and  new  forces  in  the  world  develop. 
Advances  in  technology  create  new  opportunities  that  may  require  changes 
in  organization  and  doctrine  as  well  as  hardware.  All  these  uncertainties 
lead  to  a  large  number  of  variables  that  must  be  considered.  Some  of  these 
variables  are  subject  to  our  control,  some  to  the  enemy's  and  others  to 
nobody's  control.  But  all  are  variables,  and  all  are  interdependent. 
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The  increase  in  numbers  and  kinds  of  variables  associated  with 
problems  of  the)  future  can  be  illustrated,  on  a  small  scale,  in  a  hypothetical 
study  of  a  future  weapons  system  for  an  infantry  platoon--assuming  that  the 
infantry  platoon  will  be  present  in  the  time  frame  under  consideration.  The 
variables  that  would  require  study  would  include  such  parameters  as  alter¬ 
native  weapons  systems  that  can  be  available  in  the  time  frame  under  study, 
composition  (mix)  of  kinds  of  weapons  within  the  total  system,  the  number  of 
individual  weapons  within  each  mix  of  weapons,  levels  of  warfare,  expected 
locales  of  war,  and  effects  of  supporting  weapons  of  higher  echelons.  If 
each  of  these  parameters  takes  only  three  alternative  values,  for  example, 
levels  of  warfare  to ‘be  considered  are  nuclear  warfare,  conventional  war, 
and  one  type  of  stability  of  operations,  over  700  cases  result--and  all 
significant  parameters  have  not  been  listed.  If  the  number  of  candidate 
weapons  systems  is  increased  from  three  to  six  there  are  over  1400  cases 
to  be  considered. 

It  is  in  this  environment  of  uncertainties  and  flexibility  in  use  and  inter¬ 
changeability  of  resources  (people,  dollars,  and  hardware)  that  cost-effective¬ 
ness  analysis  is  a  useful  aid.  It  assists  in  providing  increased  insight  into 
the  problem  and  as  much  relevant  information  as  possible  in  order  that  the 
decision  maker  can  concentrate  on  those  areas  where  judgment  must  be 
applied,  particularly  in  consideration  of  qualitative  aspects  and  consistency 
with  higher  echelon  considerations.  For  example,  in  a  hypothetical  force 
composition  problem  where  flexibility  in  force  composition  is  possible,  it 
has  been  determined  that  the  force  must  have  a  capability  to  destroy  certain 
kinds  of  targets  at  certain  expected  ranges.  Two  possible  alternatives  are 
artillery  and  tactical  air.  The  time  required  and  the  cost  to  destroy  these 
targets  by  use  of  each  alternative  can  be  calculated.  However,  the  importance 
of  having  a  capability  to  attack  these  targets  at  any  time  of  the  day  or  night, 
regardless  of  weather  conditions,  is  a  matter  of  judgment.  This  judgment 
can  be  better  made  when  the  cost-effectiveness  of  each  alternative  is  known, 
in  other  words,  the  price  to  be  paid  for  an  all-weather  capability  stated  in 
sufficient  detail  and  accuracy  to  be  useful  for  planning. 

The  effort  to  provide  information  so  the  commander  can  better  exercise 
his  judgment  is  also  found  in  the  estimate  of  the  situation  process.  For 
example,  a  combat  commander  can  better  apply  his  judgment  to  selection 
among  possible  courses  of  actions  when  he  has  staff  estimates--even  if 
only  rough-of  the  number  of  casualties  he  will  suffer  and  the  time  required 
to  accomplish  the  mission  for  each  of  the  proposed  courses  of  action. 

However,  the  variables  that  a  staff  estimate  must  contend  with  are  relatively 
limited.  The  friendly  organization  is  fixed,  there  is  only  one  enemy  in  only 
one  area  and  the  options  open  to  the  enemy  are  relatively  few.  For  example, 
the  enemy  can  attack,  defend,  or  execute  some  variation  of  a  withdrawal. 

For  practical  purposes,  neither  the  enemy  nor  the  friendly  force  can  introduce 
new  weapons  systems  or  change  their  fundamental  organization  or  doctrines 
in  the  time  period  covered  by  the  estimate  of  the  situation. 
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The  basis  of  cost-effectiveness  analysis  is  that  there  are  alternate  ways 
of  reaching  an  objective  and  each  alternative  requires  certain  resources  and 
produces  certain  results.  This  is  the  same  basis  of  the  estimate  of  the 
situation  which  studies  proposed  courses  of  actions,  each  of  which  requires 
certain  resources  (forces  and  supplies)  and  produces  certain  expected  results 
(time  to  take  the  objective,  casualties  incurred).  A  cost-effectiveness 
analysis  is  designed  to  examine  systematically  and  relate  costs,  effectiveness, 
and  risks  of  alternative  ways  of  accomplishing  an  objective  and  designing 
additional  alternatives  (proposed  courses  of  action)  if  those  examined  are 
found  wanting.  It  is  an  analysis  of  the  cost  and  effectiveness  of  a  system, 
such  as  a  forward  area  air  defense  or  an  air  mobile  division,  and  all  of  the 
system  implications.  It  can  be  considered  as  a  kind  of  Consumers  Research 
to  assist  in  getting  the  most  for  the  resources  to  be  expended  and  not  as  a 
search  for  the  cheapest  regardless  of  effectiveness. 

A  major  methodological  difference  between  cost-effectiveness  analyses 
and  other  military  studies  is  the  manner  in  which  the  results  are  presented. 

A  staff  study  or  a  staff  estimate,  like  a  cost-effectiveness  analysis,  considers 
sources  of  action  (alternatives).  However,  the  staff  estimate  and  staff 
study  usually  embody  a  single  recommendation  with  the  other  alternatives 
either  rarely  mentioned  or  not  as  fully  discussed  as  the  recommended  course 
of  action.  The  commander  (decision  maker)  is  given  the  full  reasoning  behind 
the  recommended  course  of  action  which  is  frequently  presented  so  that  the 
only  option  open  is  a  "yes"  or  "no"  decision. 

In  a  cost-effectiveness  analysis,  the  significant  alternatives,  the 
available  facts,  the  reasoning  process  and  the  pertinent  considerations  per¬ 
taining  to  each  significant  alternative  are  presented.  All  identifiable 
assumptions  and  data  are  presented  so  that  their  validity  can  be  questioned. 

In  addition,  and  this  is  a  major  goal  of  a  cost-effectiveness  analysis,  the 
dependence  of  the  results  of  the  analyses  on  these  assumptions  and  data 
are  tested. 

The  staff  estimate  and  staff  study  do  identify  major  assumptions. 

However,  an  implied  assumption  is  often  introduced  when  several  different 
courses  of  action  are  open  and  a  decision  is  made  to  proceed  in  one  direction. 
Such  a  decision  is  then  accepted  as  a  known  quantity  when,  in  reality,  it 
really  is  an  assumption.  There  are  many  reasons  for  such  assumptions, 
but  frequently  the  result  of  the  study  or  estimate  is  not  tested  for  sensitivity 
to  such  hidden  assumptions. 

Cost-effectiveness  analysis  places  great  emphasis  on  use  of  numbers 
and  calculations  in  any  effort  to  determine  quantitative  factors  where  possible. 
CF  course,  there  are  many  aspects  of  military  activities  that  cannot  be 
reduced  to  a  quantitative  factor.  There  is  now  no  valid  way  of  assigning  a 
number  to  morale,  the  psychological  effects  cf  a  certain  military  operations, 
or  a  host  of  other  factors.  However,  it  is  possible  to  calculate  the  number 
of  155-mm  howitzer  rounds  and  total  cost  required  to  destroy  a  certain  type 
of  target.  The  impact  of  factors  such  as  morale,  training,  reliability  of 
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allies  cannot  yet  be  calculated  and  are  now  matters  of  intuitive  judgment. 

A  cost-effectiveness  analysis  seeks  to  quantify  what  can  be  logically 
calculated  so  that  the  decision  maker  knows  the  extent  to  which  intuitive 
judgment  must  be  used  in  making  a  decision. 

Essential  Elements 

The  essential  elements  of  a  cost-effectiveness  analysis  are: 

1.  Objective(s)  (functions  to  be  accomplished). 

2.  Alternatives  (feasible  ways  of  achieving  the 
desired  military  capability  or  accomplishing 
the  function). 

3.  Cost  of  resources  required  by  each  alternative. 

4.  A  set  of  mathematical  or  logical  relationships 
among  the  objectives,  alternatives,  environment 
and  resources  (models). 

5.  A  criterion  for  choosing  the  preferred  alternative. 

The  Objective 

The  determination  of  the  objective  is  often  complex.  In  order  to  design 
alternatives  properly,  the  problem  must  be  analyzed  to  determine  the  real 
functional  need  underlying  the  requirements  for  certain  organizations  and 
hardware  systems.  Thorough  examination  of  the  functional  need  usually 
brings  insight  into  the  problem  and  leads  to  generating  alternatives  that 
may  accomplish  the  desired  goal.  Close  examination  of  objectives  stated 
only  in  terms  of  specific  organizations  or  systems  often  discloses  that  the 
net  result  is  not  a  significantly  new  or  improved  capability  but  a  relatively 
minor  product  improvement.  This  does  not  imply  that  product  improvements 
are  not  needed  but  rather  that  a  full  understanding  of  the  true  significance 
of  what  is  being  proposed  for  purchase  is  necessary.  For  example,  in 
stating  a  requirement  for  an  artillery  system  with  a  specified  minimum  range 
capability,  the  real  objective  may  be  a  capability  to  destroy  certain  kinds 
of  targets  under  certain  conditions.  By  examining  the  problem  from  the 
functional  basis,  the  planner  is  better  able  to  understand  the  problem.  This 
insight  may  lead  to  other  alternatives  that  should  be  studied.  The  examina¬ 
tion  may  show  that  the  proposed  new  artillery  system  is  only  one  alternative 
to  accomplishing  the  objective  and  that  another  alternative  may  be  preferable. 

There  are  practical  limits  on  the  definition  of  the  objective.  Every 
military  activity  is  part  of  a  larger  activity  and  it  is  necessary  to  draw  the 
line  at  some  point.  However,  the  objective  should  not  be  unduly  restricted 
by  confusion  with  performance  characteristics  such  as  speeds,  weights, 
muzzle  velocities,  hit-kill  probabilities,  and  so  forth. 
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Alternatives 


In  military  planning  there  is  rarely  only  one  exclusive  way  of  achieving 
a  given  objective.  Each  way  has  its  own  price  tag  of  time,  men,  facilities, 
materiel,  and  money.  Assume,  for  example,  that  the  planning  problem  — 
admittedly  over-simplified —  is  to  design  a  new  type  of  division  with  certain 
capabilities.  In  satisfying  these  capabilities,  the  TOE  designer  has  many 
alternatives.  For  the  same  capability,  is  it  better  to  have  more  mobility 
(trucks,  aircraft,  and  other  vehicles)  and  less  manpower,  or  perhaps  more 
mortars  and  fewer  riflemen?  The  alternatives  are  limited  only  by  creative 
imagination  and  good  military  judgment.  By  exploring  alternative  ways  of 
using  resources  it  is  often  possible  to  discover  ways  of  achieving  an  ob¬ 
jective  with  fewer  resources,  or  accomplishing  more  with  the  same  resources. 
All  feasible  and  significant  capabilities  to  accomplish  the  objective  should 
be  considered,  including  the  capabilities  of  the  Navy,  Air  Force,  and  Marines. 
Prejudices,  "party-line"  and  other  forms  of  preconceived  notions  should  be 
avoided  in  the  design  of  alternatives. 

cost 


Determining  the  cost  of  each  alternative  is  based  on  incremental  costs. 
These  are  the  net  costs  of  adopting  the  alternative.  Such  costs  are  deter¬ 
mined  after  due  allowances  for  those  resources  already  paid  for  regardless 
of  whether  the  alternative  is  adopted,  and  would  be  available  for  use  under 
the  alternative  if  it  were  adopted.  In  determining  the  cost  of  an  alternative 
all  the  resource  implications  are  considered.  The  alternative  is  treated  in 
a  system  context.  For  example,  the  cost,  admittedly  oversimplified,  cf 
adopting  a  new  radio  would  include  not  only  the  cost  of  the  radio  and  its 
development,  but  also  the  costs  of  training  people  to  operate  it,  the  total 
cost  of  maintaining  the  radios,  and  the  cost  of  the  additional  radios  required 
for  maintenance  float,  replacement,  combat  consumption,  and  so  forth. 

Costs  need  not  be  stated  in  precise  terms  down  to  the  last  dollar  or  man. 
However,  the  costs  must  be  accurate  enough  to  permit  evaluating  the 
military  worth  (effectiveness)  together  with  the  costs.  Like  everything  else, 
this  rule  must  be  applied  with  discretion.  In  dealing  with  systems  way  out 
in  the  future  die  accuracy  of  the  cost  estimate,  whether  it  is  an  absolute 
figure  or  a  range,  probably  is  inverse  to  the  distanceout  in  the  future. 
Usually  cost  estimates  are  tested  by  sensitivity  analysis.  These  are  re¬ 
petitive  analyses  using  different  quantitative  values  to  determine  if  the 
results  are  sensitive  to  the  values  assigned.  Such  analyses  give  the 
decision  maker  a  better  understanding  of  how  much  uncertainty  is  involved 
if  there  are  significant  errors  in  the  cost  estimates.  He  can  then  better 
judge  if  the  investment  is  worth  the  payoff  considering  the  uncertainties 
involved. 

Models 

Models  are  used  in  cost-effectiveness  analysis  to  cope  with  the. host  of 
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variables  that  are  inherent  in  problems  cf  the  future.  A  model  is  simply 
certain  relationships  expressed  in  some  way  to  simulate  real  or  expected 
conditions  in  order  to  foresee,  even  to  a  limited  extent,  the  expected  out¬ 
come  of  a  course  of  action.  Models  assist  in  simplifying  the  problem,  in 
identifying  the  significant  components  and  interrelations,  in  determining 
which  variables  are  especially  important  for  the  decision  at  issue,  and  which 
variables  can  be  suppressed.  In  this  manner,  the  decision  process  can  be 
more  precisely  focussed  on  those  areas  which  require  a  judgment  decision. 

Models  range  from  simple  graphs  to  complex  equations  and  can  also  take 
the  form  of  a  wargame  or  field  maneuver.  The  estimate  of  the  situation  and 
staff  estimates  also  use  models.  The  comparison  of  proposed  courses  of 
action  against  enemy  capabilities  or  expected  difficulties  and  the  comparison 
among  the  proposed  courses  of  action  repTesent  uses  of  models  to  foresee  the 
future  outcome  of  an  action. 

All  models,  simple  or  complex,  are  abstractions  of  the  real  world  and 
their  validity  depends  on  the  proper  selection  of  assumptions  and  the 
correctness  of  the  relations  portrayed,  and  the  pertinence  of  the  factors 
included  in  the  model.  Two  aspects  of  model  building  are  particularly 
troublesome,  quantification  and  the  treatment  of  uncertainty.  Some  variables 
are  difficult  to  quantify,  such  as  the  continued  availability  cf  certain  support 
from  an  ally.  This  difficulty  leads  either  to  the  neglect  cf  such  variables 
by  ignoring  them  or  by  a  qualitative  modification  of  a  solution  derived  from 
the  treatment  of  other  variables  that  have  been  properly  quantified.  Such 
treatment  often  results  in  the  difficult-to-quantify  variables  being  lost  within 
all  the  other  qualitative  considerations  that  must  be  weighed  when  the  time 
comes  to  recommend  action  on  the  basis  of  the  solution  from  the  model. 

The  influence  of  the  variable  that  cannot  be  quantified  and  all  uncertain¬ 
ties  must  be  specifically  addressed  in  the  model  unless  it  can  be  demonstrated 
by  logic  or  analysis  that  they  are  trivial,  affect  all  alternatives  roughly  the 
same,  or  the  results  are  insensitive  to  them.  Guessing  may  lead  to  disaster. 
For  example,  if  there  is  uncertainty  about  8  factors,  a  best  guess  might  be 
made  on  each  of  them.  If  there  is  a  60%  probability  that  each  best  guess  is 
right,  then  the  probability  that  all  guesses  are  right  is  less  than  2%.  Relying 
on  best  guesses,  in  this  case,  would  be  ignoring  all  the  outcomes  with  more 
than  98% probability  of  occurring.  Uncertainties  and  the  problem  cf  the 
factors  that  cannot  be  quantified  can  be  handled  through  various  techniques 
such  as  Monte  Carlo  sampling,  contingency  analysis,  (see  Glossary)  and 
even  wargaming  for  certain  purposes. 

Models  that  portray  relations  incorrectly  also  lead  to  false  results. 

For  example,  some  models  are  based  on  the  persistence  principle  which 
states  that  what  is  happening  or  has  happened  will  persist.  This  type  of 
model  is  dangerous  except  for  very  short-term  uses.  For  example,  .it  is 
wrong  to  assume  that  the  enemy  tactics  used  during  the  Korean  war  will 
continue  to  be  used  in  the  future  against  new  types  cf  equipment  and  tactics 
that  may  be  introduced.  Some  models  depend  on  extrapolation  which  assumes 
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that  trends  will  continue  uninterrupted.  Such  models  lend  themselves 
readily  to  mathematical  treatment  but  are  often  erroneous  because  of 
failure  to  consider  what  is  called  the  Law  of  Diminishing  Returns.  For 
example,  a  machine  gun  can  fire  at  a  certain  high  rate.  However,  this 
high  rate  cannot  be  maintained  for  very  long  (extrapolation)  because  the 
barrel  would  soon  be  burned  out. 

Models  can  be  classified  into  two  general  types--exact  (deterministic) 
or  probabilistic.  An  exact  model  of  warfare,  of  course,  is  impossible  in 
peacetime.  However,  it  is  possible  to  create  an  almost  exact  model  of 
some  specific  piece  of  hardware  or  activity  and  subject  it  to  test.  The 
final  product  of  the  model  will  then  closely  approximate  the  results  from 
the  actual  hardware  or  activity.  March  graphs  used  for  planning  adminis¬ 
trative  movements  are  examples  of  deterministic  models.  Most  military 
problems  are,  by  nature,  made  up  of  uncertainties.  Consequently,  they 
are  considered  as  probabilistic  when  the  uncertainty  is  identified  by  a 
probability  factor.  For  example,  a  wargame  using  a  certain  kill  probability 
for  an  air  defense  system  is  a  probabilistic  model. 

The  construction  of  models  to  evaluate  effectiveness  is  often  difficult. 
The  difficulties  arise  in  selection  cf  criteria  of  effectiveness.  It  is 
relatively  easy  to  measure  the  comparative  effectiveness  of  two  similar 
pieces  of  equipment  designed  to  accomplish  the  same  general  objective  as, 
for  example,  in  comparison  of  a  towed  I05-mm  and  a  self-propelled  105-mm 
howitzer.  However,  it  is  more  difficult  to  compare  the  effectiveness  of 
general  purpose  force  organizations  such  as  two  different  kinds  of  divisions 
or  even  two  equal-strength  infantry  battalions  having  the  same  general  kinds 
of  weapons  but  one  having  three  rifle  companies  and  the  other  having  four. 
The  impact  on  effectiveness  of  intangibles  such  as  morale  and  leadership 
can  hardly  be  calculated  and  requires  the  application  of  judgment.  Each 
study  virtually  requires  a  consideration  of  its  own  criteria  of  effectiveness. 

Models  used  in  cost-effectiveness  analysis  sometimes  tend  to  become 
mathematical  and  abstract.  Consequently,  they  may  be  difficult  to  under¬ 
stand.  A  good  cost-effectiveness  analysis  strikes  a  balance  in  the  use  cf 
models  between  simplicity  and  retention  of  enough  detail  to  ensure  that 
the  expected  outcome  of  an  expected  action  will  be  adequately  portrayed. 

In  any  case  all  models  have  certain  common  elements.  These  are  broadly 
stated  as  a  definition  of  the  problem, principal  factors  or  constraints, 
verification  and  the  decision  process —  or  application  of  criteria.  The 
validity  of  conceptual  or  mathematical  models  cannot  be  verified  in  a  cost- 
effectiveness  analysis  by  controlled  experiments.  At  the  best,  they  can  be 
tested  by  their  workability.  Questions  37  to  44  in  the  next  chapter  are 
designed  to  assist  a  review  to  test  the  workability  of  models  used  in  cost- 
effectiveness  analyses. 

Criteria 

The  most  widely  used  criteria  in  Army  studies  for  selecting  the  pre- 
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ferred  alternative  are  usually  based  on  either  equal  cost  or  equal  effective¬ 
ness  of  the  alternatives.  Another  method,  known  as  incremental  effectiveness 
at  incremental  cost,  is  used  in  special  cases.  In  the  equal  cost  form  it 
is  assumed  that  there  is  an  arbitrary  fixed  budget  or  series  of  fixed  budgets, 
and  the  analysis  determines  which  alternative  gives  the  greatest  effectiveness 
for  the  same  expenditures  or  resources.  In  the  equal  effectiveness  form,  a 
specified  and  measurable  military  effectiveness  (capability)  is  stated  and 
the  analysis  is  to  determine  which  alternative  achieves  this  effectiveness 
at  least  cost.  The  incremental  effectiveness  at  incremental  cost  method 
relates  the  increase  in  effectiveness  achieved  to  the  associated  increase  in 
resources  involved.  This  method  is  normally  used  only  as  a  last  resort 
when  neither  costs  nor  effectiveness  of  alternatives  can  be  made  equal,  e.  g. , 
when  a  capability  based  on  a  new  technology  is  to  be  added  to  the  force  and 
this  new  capability  cannot  be  approximated  by  any  practicable  combination 
of  existing  materiel  and  men. 

Exile  of  Judgment 

Judgment  is  used  throughout  a  cost-effectiveness  analysis  in  the  same 
manner  as  in  the  making  of  an  estimate  of  the  situation  or  a  staff  estimate. 
Judgment  is  used  in  analyzing  the  objective,  deciding  which  alternatives 
(courses  of  action)  to  consider,  which  factors  are  relevant  and  the  inter¬ 
relations  among  these  factors,  which  numerical  values  are  to  be  used,  and 
in  analyzing  and  interpreting  the  results  of  the  analysis.  The  goal  of  a 
cost-effectiveness  analysis  is  to  keep  all  judgments  in  plain  view  and  to 
make  clear  the  logic  used.  It  also  shows  the  sensitivity  of  the  results  to 
the  significant  judgments  made.  The  depth  of  a  cost-effectiveness  analysis 
is  tempered  by  the  time  and  manpower  available  and  the  importance  cf  the 
subject  matter,  A  cost-effectiveness  analysis  requires  resources  and  it 
must  serve  as  an  aid  to  the  making  of  decisions  and  not  be  a  mere  intellectual 
exercise , 

Review  of  Studies 


There  are  probably  almost  as  many  different  ways  of  reviewing  a  study 
containing  cost-effectiveness  analysis  as  there  are  reviewers.  Furthermore, 
the  time  available  for  review  is  variable  and  studies  lack  a  common  format. 

It  is  suggested  that  the  points  listed  below  be  checked  specifically  in  the 
early  stages  of  a  review. 

a.  Statement  of  criteria  used  to  judge  effectiveness. 

b.  Statement  of  criterion  used  to  select  preferred 
alternative , 

c.  Use  of  incremental  costs. 

d.  Explanation  of  logic  of  models. 
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e.  Presence  or  lack  of  analysis  of  sensitivity  of  the 
results  to  significant  data  and  assumptions. 

Without -these  elements  being  present,  the  study  will  probably  be 
of  poor  quality. 

Army — conducted  studies  containing  cost-effectiveness  analysis 
usually  do  not  have  a  uniform  organizational  pattern  but  many  generally 
follow  the  Staff  Study  format.  Qithat  basis,  the  key  questions  in  the  next 
chapter  have  been  grouped  under  these  headings: 

Statement  of  the  Problem 

Assumptions 

Alternatives 

Documentation 

cost 

Relationships  (Models) 

Effectiveness 

Criteria 

Conclusions  and  Recommendations 

The  grouping  under  the  above  headings  inevitably  leads  to  some  dupli¬ 
cation  of  material,  particularly  on  the  use  of  analytical  tools  such  as  sen¬ 
sitivity  and  contingency  analysis,  "his  duplication  has  been  kept  to  a 
minimum  but  full  coverage  has  been  retained  under  each  heading  as  a 
convenience  to  the  reviewer  who  wishes  to  refer  to  a  specific  heading. 

The  Glossary  is  designed  to  give  a  non-technical  definition  of  terms 
frequently  used  in  cost-effectiveness  analyses. 

The  annotated  Bibliography  has  been  designed  for  the  reviewer  who 
desires  to  read  further  into  the  methodology  of  cost-effectiveness  analysis. 
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Chapter  11 
KEY  QUESTIONS* 
STATEMENT  OF  THE  PROBLEM 


1.  IS  THE  PROBLEM  STATED  THE  REAL  PROBLEM? 

An  improper  statement  of  the  problem  often  results  in  either  studying 
the  wrong  problem  or  precluding  consideration  of  worthy  alternatives  . 
These  defects  are  usually  avoided  by  a  statement  of  the  problem  in  terms 
of  a  functional  need--the  job(s)  to  be  done — without  implying  how  it  is  to  be 
done.  A  statement  of  the  problem  in  terms  of  requirements  for  kinds  of 
forces,  systems,  or  performance  characteristics,  except  if  it  is  a 
follow-on  to  a  previously  approved  study  of  a  functional  need,  should  be 
critically  examined  to  ensure  that  the  wrong  problem  is  not  being  studied 
and  that  worthy  alternatives  are  not  automatically  excluded  from 
consideration.  For  example,  although  the  stated  problem  (no  previous 
study  of  functional  need)  may  be  to  select  a  rifle  to  meet  certain  capabili¬ 
ties  (requirement  statement),  the  real  problem  might  be  providing  the 
rifle  squad  with  adequate  firepower  to  accomplish  certain  functions  (func¬ 
tional  need).  In  such  a  case,  a  rifle  is  only  one  possible  alternative. 

A  word  of  caution  is  in  order.  There  often  is  a  practical  limit  on  the 
depth  of  the  statement  of  the  functional  need  or  the  study  may  become 
unmanageable.  For  example,  in  the  case  cited  the  functional  need  could 
be  conceivably  so  stated  that  the  rifle  squad  itself  becomes  only  one  altern¬ 
ative  to  solving  a  larger  problem.  To  avoid  this  difficulty,  either  certain 
broader  decisions  must  be  considered  as  made,  thereby  narrowing  the 
scope  of  the  study,  or  the  broader  study  undertaken.  When  the  former 
approach  is  taken,  the  study  is  known  as  a  suboptimization.  The  reviewer, 
based  on  his  knowledge  and  judgment,  must  determine  if  the  suboptimiza¬ 
tion  has  so  narrowed  the  scope  of  the  problem  that  the  real  problem  has 
been  missed  or  worthwhile  alternatives  excluded. 


2.  DOES  THE  STUDY  IDENTIFY  IMPLIED  SIGNIFICANT  COMPONENTS 
OF  THE  PROBLEM  THAT  MUST  BE  FULLY  TREATED  IN  THE 
STUDY? 

Like  the  mission  statement  in  an  estimate  of  the  situation,  theproblem 
to  be  treated  in  a  cost-effectiveness  analysis  must  be  analyzed  to  identify 
all  functions  that  must  be  performed.  Some  of  these  implied  functions  are 


*Those  questions  that  are  considered  particularly  important  and  of  widest 
application  have  been  underlined. 
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often  not  apparent  at  first.  The  reviewer  should  watch  for  implied  signifi¬ 
cant  component  parts  of  the  problem  that  are  neither  identified  nor  treated 
fully  in  the  study.  The  reviewer  should  also  watch  for  other  problems 
that  are  opened  up  or  revealed  by  the  study  that  should  be  further  investi¬ 
gated. 
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ASSUMPTIONS 


3.  ARE  ALL  ASSUMPTIONS  IDENTIFIED? 

The  reviewer  should  watch  for  assumptions  that  are  not  identified  as 
such  because  assumptions  imply  a  limitation  or  a  judgment.  In  order  to 
evaluate  the  study  properly,  it  is  necessary  to  assess  the  impact  of  the 
limitations  and  the  validity  of  the  judgments  contained  in  all  the  assump¬ 
tions.  An  example  of  a  common  assumption  that  is  often  not  identified  is 
that  a  given  unit  operates  by  itself  .Asa  result,  in  measuring  the 
effectiveness  of  a  division,  for  example,  inadequate  consideration  is 
sometimes  given  to  the  support  the  division  receives  from  non-divisional 
units  such  as  corps  artillery  or  tactical  air  units.  This  failure  to  consider 
non-divisional  support  may  lead  to  erroneous  conclusions  and  recommen¬ 
dations.  Another  frequently  hidden  assumptiort  is  that  the  enemy'  s  doctrine 
and  tactics  are  rigid  although  ours  are  flexible. 


4.  ARE  THE  ASSUMPTIONS  UNDULY  RESTRICTIVE? 

Assumptions  are  properly  used  to  narrow  the  scope  of  the  study  to 
manageable  proportions.  However,  the  assumptions  should  be  examined 
to  determine  whether  they  unduly  restrict  the  study  by  eliminating  possible 
significant  alternatives  or  by  narrowing  the  scope  of  consideration  to  the 
point  that  the  conclusions  and  recommendations  may  be  in  error.  This 
examination  may  be  required  throughout  the  review  of  the  study  and  not 
only  during  the  review  of  the  stated  assumptions. 

Assumptions  covering  the  subjects  listed  below  often  unduly  restrict 
the  scope  of  the  study  and  lead  to  questionable  conclusions  and  recommen¬ 
dations. 


a.  Non-availability  or  limited  availability  of 
support  from  other  services  (e.g., 
tactical  air  support  or  MATS  effort). 

b.  Locale  of  operations. 

c.  Duration  and  intensity  of  operations. 

d.  Enemy  organization,  operations,  and 
reactions  to  our  decisions. 

e.  Time  period  covered. 
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5.  DO  ANY  OF  THE  MAJOR  ASSUMPTIONS  UNJUSTIFIABLY  TREAT 
QUANTITATIVE  UNCERTAINTIES  AS  FACTS? 

An  uncertainty  can  be  defined  as  the  lack  of  definitive  knowledge  for 
assigning  values  or  probabilities  to  factors  that  influence  decisions  . 
Uncertainties  can  be  either  quantitative  (risks)  or  qualitative.  (See 
UNCERTAINTY  and  RISK  in  Glossary.)  Examples  of  quantitative  uncertain¬ 
ties  are  hit-kill  probabilities,  equipment  availability  rates,  ammunition 
expenditure  rates,  and  reliability  statements.  The  availability  of  base 
rights  in  a  foreign  country  at  some  future  time,  or  the  start  of  aggression 
by  the  potential  enemy  in  a  given  year  are  examples  of  a  qualitative  un¬ 
certainty.  (See  next  question.) 

The  reviewer  should  be  alert  for  stated  and  implied  major  assump¬ 
tions  that  assign  fixed  values  to  quantitative  uncertainties  and  then  treat 
these  estimates  as  facts.  A  common  example  is  the  assumption  that  a 
proposed  weapon  system  will  have  a  certain  hit-kill  probability.  It  is  often 
better  to  handle  significant  uncertainties  by  sensitivity  analysis.  This  is 
a  repetitive  analysis  using  different  quantitative  values  to  determine  if  the 
results  are  sensitive  to  the  values  assigned.  When  significant  uncertain¬ 
ties  are  treated  as  facts  by  assumption,  the  conclusions  and  recommenda¬ 
tions  cf  the  study  may  be  no  more  valid  than  the  assumption  unless  it  can 
be  demonstrated  that  the  conclusions  and  recommendations  are  not 
sensitive  to  plausible  errors  in  the  "  facts.  " 

The  number  of  sensitivity  analyses  required,  and  feasible,  is  a 
matter  of  judgment.  There  are  limits  to  the  time  and  manpower  available 
for  a  given  study.  Sometimes  an  educated  guess,  considering  all  the 
circumstances,  will  suffice.  In  effect,  the  reviewer  must  judge  whether 
the  study  agency  has  performed  adequate  sensitivity  analyses  considering 
all  the  circumstances,  the  importance  of  the  subject,  and  whether  further 
sensitivity  analysis  may  significantly  affect  the  conclusions  and 
recommendations. 


6.  DO  ANY  OF  THE  MAJOR  ASSUMPTIONS  TREAT  QUALJTATIVE 
UNCERTAINTIES  AS  FACTS? 

Major  qualitative  uncertainties  treated  as  assumptions  also  tend  to 
dictate  conclusions.  A  common  qualitative  uncertainty  that  may  dictate 
the  conclusions  concerns  the  estimate  of  the  enemy.  Many  studies  are 
based  on  intelligence  estimates,  or  target  arrays  prepared  or  approved 
by  the  Defense  Intelligence  Agency.  However,  these  estimates  are  some¬ 
times  assumed  to  be  facts.  In  such  cases,  this  often  results  in  the  enemy 
being  considered  to  be  inflexible  and  no  allowances  are  made  for  him  to 
react  in  different  ways  to  our  operations  or  to  our  introduction  of  new 
capabilities.  When  it  is  not  definitely  known  how  we  will  operate  or  be 
equipped  10 years  hence,  it  is  questionable  to  assume  that  the  enemy 
operations  and  equipment  in  the  future  can  be  predicted  with  certainty  . 
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Other  qualitative  uncertainties,  stated  or  implied,  that  should  be 
treated  with  caution  are  those  associated  with  political  considerations. 
Examples  are  availability  of  base  rights,  assurance  cf  overflight  per¬ 
mission,  and  composition  of  political  and  military  alliances  on  either  side. 

Treatment  of  the  kinds  cf  uncertainties  discussed  above  in  an  analysis 
is  not  simple,  but  the  effects  cf  such  uncertainties  on  the  conclusions  should 
not  be  neglected  in  a  study.  One  method  to  cope  with  significant  uncertain¬ 
ties  of  this  kind  is  to  use  contingency  analysis.  This  involves  repetitive 
analysis  with  different  qualitative  assumptions,  such  as  type  cf  conflict  or 
enemy  capabilities,  to  determine  their  effects  for  comparison  with  the 
results  cf  the  initial  analysis.  The  amount  cf  contingency  analysis  required 
has  to  be  a  matter  of  judgment,  as  discussed  in  the  previous  question. 


7.  ARE  THE  MAJOR  ASSUMPTIONS  REASONABLE? 

Major  assumptions  should  also  be  tested  to  determine  if  they  are 
reasonable.  This  test  can  be  facilitated  if  the  study  documents  or  provides 
some  explanation  of  why  each  assumption  was  made  so  that  the  reasons  can 
be  evaluated  by  the  reviewer.  A  useful  technique  for  reviewers  is  to  try 
to  think  cf  other  major  assumptions  that  are  plausible.  If  these  invalidate 
the  conclusions  and  recommendations,  then  the  study  is  questionable. 
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ALTERNATIVES 


8 .  ARE  CURRENT  CAPABILITIES  ADEQUATELY  CONSIDERED  AMONG 
THE  ALTERNATIVES? 

Current  capabilities  should  not  be  omitted  from  consideration  in 
construction  of  alternatives  except  for  valid  reasons  that  are  clearly 
stated.  Valid  reasons  may  include  failure  cf  the  current  system  to  accom¬ 
plish  the  current  mission,  or  a  significant  degradation  of  capability  relative 
to  that  of  the  potential  enemy.  Consideration  of  current  capabilities  is  an 
improvement  that  is  worth  the  expenditure  of  new  resources.  By  consider¬ 
ing  current  capabilities,  much  of  whose  costs  are  already  paid  for,  as  an 
alternative,  the  study  can  show  the  difference  in  effectiveness  and  costs 
that  result  from  the  adoption  of  the  proposed  new  system  or  organization. 
(See  question  23.)  Current  capabilities  should  also  be  considered,  where 
appropriate,  as  a  component  of  an  alternative. 


9.  ARE  "  TRADE-OFFS”  WITH  EXISTING  SYSTEMS  CR  ORGANIZATIONS 
ADEQUATELY  CONSIDERED  WITHIN  THE  ALTERNATIVES? 

Where  appropriate,  the  design  of  alternatives  should  consider 
"  trade-offs"  with  existing  systems  or  organizations.  Possible  examples 
are:  (1)  in  studying  the  increased  use  of  Army  transport  aircraft  an 
alternative  might  include  reduction  in  other  means  of  transport;  (2)  in  a 
study  on  an  improved  fire  control  system  an  alternative  might  include  a 
reduction  in  ammunition  stockage. 


10.  ARE  THE  APPROPRIATE  CAPABILITIES  OF  THE  AIR  FORCE,  NAVY, 

OR  MARINE  CORPS  CONSIDERED  AMONG  THE  ALTERNATIVES? 

The  alternatives  should  consider  the  capabilities  of  Air  Force,  Navy, 
or  Marine  Corps  as  appropriate.  The  Army  usually  conducts  combat 
operations  with  the  support  of  one  or  more  of  the  other  Services  and  the 
other  Services  are  charged  by  law  with  furnishing  certain  support  to  the 
Army.  These  types  of  supports  are  listed  in  JCS  Publication  2  (UNAAF). 

For  example,  CONUS  air  defense  is  not  the  exclusive  responsibility  of 
either  the  Air  Force  or  the  Army.  A  CONUS  air  defense  problem  must 
consider  Army  surface-to-air  missiles.  Air  Force  manned  interceptors, 
and  Air  Force  surface-to-air  missiles. 

Current  and  projected  capabilities  of  the  other  Services  can  be 
obtained  from  a  number  of  different  sources  including  the  Five-Year  Force 
Structure  and  Financial  Plan  maintained  by  each  Service.  The  reviewer 
should  bear  in  mind  that  functions  such  as  air  defense,  the  attack  of  sur¬ 
face  targets,  reconnaissance  in  the  vicinity  cf  the  FEBA,  and  transportation 
within  a  theater  are  not  the  exclusive  responsibilities  of  the  Army. 
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1 1  ARE  MIXTURES  OF  SYSTEMS  (ORGANIZATIONS)  CONSIDERED 

AMONG  THE  ALTERNATIVES? 

The  reviewer  should  watch  for  failure  to  consider  appropriate  alterna¬ 
tives  tKat  are  based  on  mixtures  of  two  or  more  systems  (organizations)to 
combine  the  best  features  of  each.  For  example,  in  comparing  certain 
transportation  systems  one  alternative  for  surface  transportation  might  be 
a  combination  of  truck,  rail  and  water  systems  rather  than  only  a  truck 
system.  In  the  same  manner,  the  study  of  a  proposed  new  missile  system 
might  consider  as  an  alternative  a  suitable  combination  cf  existing  missile 
and  gun  systems  and  aerial  fire  support  rather  than  only  an  existing  missile 
system. 


12.  ARE  ANY  FEASIBLE  AND  SIGNIFICANT  ALTERNATIVES  OMITTED? 

A  major  contribution  that  a  reviewer  can  make  is  to  point  out  signifi¬ 
cant  and  feasible  alternatives  that  the  study  may  have  failed  to  consider. 

If  any  of  the  answers  to  the  previous  questions  on  "  Alternatives"  are  in 
the  negative  then  it  is  possible  that  some  feasible  and  significant  alterna¬ 
tives  were  not  considered.  However,  the  reviewer  must  exercise  judgment 
before  criticizing  a  study  for  failure  to  consider  all  possible  alternatives. 
There  are  practical  limits  on  the  time  and  manpower  available  for  a  given 
study.  The  relative  importance  of  the  decision  on  the  subject  under  study 
will  also  influence  the  number  of  alternatives  examined.  The  reviewer 
should  consider  these  aspects  in  determining  whether  feasible  and  signi¬ 
ficant  alternatives  have  been  omitted  to  the  detriment  of  arriving  at  sound 
recommendations  . 

On  the  other  hand,  a  large  number  of  alternatives  may  only  indicate 
that  minor  variations  have  been  considered  as  new  alternatives.  Excessive 
use  of  such  minor  variations  as  alternatives  often  beclouds  significant 
choices. 
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DOCUMENTATION  AND  DATA 


13.  IS  THE  STUDY  ADEQUATELY  DOCUMENTED? 

A  key  element  of  systematic  analysis  is  sufficient  documentation  of 
methods  and  sources  so  that  with  the  same  material,  other  study  groups 
can  arrive  at  substantially  the  same  results.  Without  such  documentation, 
a  study  appeals  for  acceptance  solely  on  faith  in  the  authority  and  expertise 
of  the  study  group  and  without  critical  examination  of  the  sources  and 
methods  used  to  arrive  at  the  recommendations. 

The  test  of  adequacy  can  be  applied  by  examining  the  models,  data, 
assumptions,  etc.,  to  determine  if  they  are  stated  in  such  a  way  that 
another  study  agency  could  trace  through  the  steps  of  the  study  and  arrive 
at  substantially  the  same  results  and  conclusions.  A  study  that  is  not 
adequately  documented  will  usually  fare  poorly  when  reviewed  by  agencies 
lacking  the  detailed  knowledge  of  the  pqoblem  that  can  sometimes  compen¬ 
sate  for  poor  documentation.  Inadequately 'documented  studies  may 
require  only  slight  additions  to  be  properly  documented. 


14.  ARE  THE  FACTS  STATED  CORRECT? 

It  is  usually  neither  possible  nor  necessary  for  a  reviewer  to  verify 
all  the  factual  material  presented  in  a  study,  but  it  is  advisable  to  spot 
check.  Particular  attention  should  be  paid,  where  possible,  to  the  factual 
material  on  which  conclusions  and  recommendations  depend.  If  many 
errors  are  involved  then  a  thorough  verification  of  the  facts  presented  may 
be  in  order. 

In  reviewing  factual  material,  its  source  should  be  examined  critically. 
For  example,  frequent  use  is  made  of  data  contained  in  FM  101-10, 

"  Organization,  Technical,  and  Logistical  Data"  and  similar  publications. 
The  data  contained  in  these  manuals  are  usually  averages  cf  historical  data 
obtained  from  certain  kinds  of  operations  in  specific  theaters.  The  un¬ 
questioning  use  of  these  average  figures  may  lead  to  erroneous  conclusions 
because  the  use  cf  an  average  hides  significant  variations  that  exist  in  the 
real  world.  A  tank  battalion  does  not  always  cover  the  same  number  cf 
miles  each  day  even  over  the  same  terrain.  Further,  the  data  contained 
in  the  reference  manuals  may  not  have  been  computed  for  the  purpose  re¬ 
quired  in  the  study  and  considerations  important  to  the  study  may  not  be 
included  in  the  calculations.  For  example,  ammunition  expenditure  rates 
contained  inFM  101- 10  are  based  on  World  War  II  and  Korean  experience 
and  organizations.  The  use  of  these  rates  for  projected  operations  in  the 
1965-70  time  frame  would  be  questionable. 

Projection  of  current  operational  experiences  into  future  time  frames 
should  also  be  examined  critically.  For  example,  a  study  used  as  data 
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that  an  armed  helicopter'  s  missions  are  A%  escort,  B%  casualty  production, 
and  the  remaining  missions  for  suppressive  fire.  These  data  were  obtained 
from  experience  in  Viet  Nam  operations.  This  unquestioning  projection  cf 
such  data  into  future  operations  in  other  areas  fails  to  allow  for  possible 
introduction  of  significantly  new  US  and  enemy  tactics  and  may  result  in 
conclusions  and  recommendations  on  how  better  to  cope  with  the  last  war. 


15.  ARE  THE  FACTS  STATED  WITH  PROPER  QUALIFICATION? 

In  addition  to  checking  the  validity  of  the  factual  material,  it  is  good 
practice  to  check  the  factual  material  for  completeness.  Some  material 
may  be  factually  correct  in  isolation  but  may  take  on  a  different  signifi¬ 
cance  when  other  facts  are  added.  Fqr  example,  it  is  true  that  infantry 
units  can  march  at  an  average  rate  cf  2.5  miles  per  hour  .  However,  this 
rate  is  valid  only  on  relatively  level  roads. 


16.  ARE  FINDINGS  AND  DATA  FROM  FIELD  EXERCISES  AND  FIELD 

TESTS  USED? 

Field  exercises  and  field  tests  can,be  excellent  sources  for  effective¬ 
ness  data.  When  used  in  a  study,  such  data  should  be  carefully  examined. 
The  reviewer  should  determine  whether  the  data  were  obtained  by  measure¬ 
ments  or  by  judgment  of  individuals  and  if  similar  data  would  likely  be 
obtained  if  the  field  test  or  field  exercise  were  conducted  again  by  another 
agency  or  unit.  The  circumstances  surrounding  the  field  exercise  or  field 
test  should  be  reviewed,  where  possible,  to  determine  if  any  artificialities 
(there  are  always  some  in  any  peacetime  operation)  were  of  sufficient  in¬ 
fluence  to  affect  the  results  cf  the  study  based  on  field  data.  Field 
exercises  usually  have  many  parameters  and  very  few  runs,  therefore 
making  it  very  difficult  to  single  out  cause  and  effect. 

Common  artificialities  that  may  significantly  affect  data  from  field 
exercises  and  field  tests  include: 

a.  Inability  to  assess  effectiveness  of  air 
defense  fires,  air-to-surface  fires,  and 
ground-to-ground  fires  . 

b.  Lack  of  realistic  levels  of  support 
from  the  other  Services  or  other 
supporting  units  .  Often  this  support 
is  either  not  available  or  available 
in  abnormally  large  amounts. 

c.  Use  of  administrative  breaks  for  rests, 
intensive  resupply,  and  maintenance 
operations  . 
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d.  Unrealistic  maneuver  and  deployment 
because  of  restricted  maneuver  areas. 

e.  The  units  or  quantities  of  materiel 
tested  are  not  a  valid  sample  either 
because  of  inadequate  size  or  of  bias 
in  composition. 

f.  Poor  or  inadequate  reporting  of  events 
of  the  exercise. 

g.  Effect  on  actions  of  participants  caused 
by  use  of  only  blank  ammunition. 


17 .  ARE  THE  DATA  FROM  SUPPORTING  WARGAMES  VALID? 

Studies  sometimes  use  the  findings  of  wargames  as  facts.  In  eval¬ 
uating  such  facts,  the  reviewer  should  bear  in  mind  the  nature  of  wargames. 
Basically,  a  wargame  involves  a  hypothetical  situation  in  which  two 
opposing  sides  interact  in  accordance  with  a  set  of  more  or  less  definite 
rules.  In  all  forms  of  wargames,  the  play  is  determined  either  by  mech¬ 
anistic  rules  or  judgments  made  by  individuals  or  both.  These  rules  and 
judgments  are  based  on  assumed  situations  and  known  or  assumed  facts 
and  system  characteristics.  Well  planned  and  executed  wargames  are 
excellent  teaching  devices  and  provide  the  participants  with  good  insights 
into  the  problem  gamed.  Such  games,  if  well  documented,  usually  provide 
a  body  of  synthetic  data  which,  when  analyzed,  provides  clues  to  problem 
areas  that  need  further  investigation. 

In  determining  the  validity  of  the  findings  of  wargames,  the  reviewer 
should  judge  how  well  the  game  portrayed  reality  and  should  satisfy  himr 
self  on  the  validity  of  the  judgments  and  assumptions  used  in  the  conduct 
of  the  game.  The  study  should  lay  out  for  scrutiny  the  major  judgments 
and  assumptions  used  in  the  wargame.  It  is  recognized  that  it  is  usually 
not  possible  to  lay  out  all  judgments  and  assumptions  used  in  the  wargame. 
In  any  case,  the  reviewer  should  weigh  the  dependence  cf  the  conclusions 
and  recommendations  on  the  findings  of  the  wargame  and  consider  whether 
other  competent  players  playing  the  same  game  would  have  arrived  at 
similar  results. 


18.  ARE  THE  PERFORMANCE  CHARACTERISTICS  VALID? 

Performance  characteristics  data  are, often  the  key  elements  in  the 
determination  of  the  effectiveness  cf  a  system.  In  evaluating  the  validity 
of  performance  characteristics,  the  source  of  the  data  should  be  examined. 
When  the  claimed  performance  characteristics  are  essential  to  the  con¬ 
clusions  and  recommendations  and  the  source  of  the  data  is  not  clearly 
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stated,  additional  information  may  be  required  from  the  agency  that  pre¬ 
pared  the  study.  This  may  not  be  necessary  if  the  study  contains  a 
sensitivity  analysis  of  a  reasonable  range  of  values  for  the  performance 
characteristics. 

Performance  characteristics  based  on  a  manufacturer'  s  claims  are 
often  too  optimistic  .  Performance  characteristics  derived  from  tests  at 
research  installations  also  require  examination.  Sometimes,  such  per¬ 
formance  characteristics  are  derived  under  controlled  conditions  that 
neglect  the  man-machine  relation  that  exists  under  field  condition.  Even 
performance  characteristics  derived  from  field  tests  should  be  examined 
Such  tests  can,  at  times,  produce  misleading  results  due  to  artificialities 
caused  by  various  peacetime  restrictions  such  as  safety  regulations  and 
choice  of  test  areas. 

If  faced  with  questionable  performance  characteristics  that  are  key 
to  the  conclusions,  the  reviewer  should  consider:  (1)  performing  a 
sensitivity  analysis  himself  if  his  time  and  the  data  in  the  study  permit; 
(2)  requesting  validation  of  the  performance  characteristics  and 
sensitivity  analysis. 


19.  ARE  ANY  OF  THE  DATA  DERIVED  FROM  QUESTIONNAIRES? 

The  data  obtained  from  questionnaires  should  be  examined  to  determine 
the  validity  of  the  questions,  the  adequacy  of  the  sample  and  statistical 
procedures,  and  the  expertness  of  the  personnel  questioned.  For  example, 
one  study  cited  data  on  the  frequency  of  kinds  of  missions  expected  to  be 
flown  by  Army  aircraft  in  a  conventional  war.  The  data  were  based  on  a 
questionnaire  completed  by  Army  aviators  at  one  Army  post.  There  was 
no  operational  experience  applicable  to  the  study  and  an  educated  guess  or 
subjective  judgment  was  in  order.  However,  in  this  case,  the  judgment  of 
those  who  order  Army  aviation  missions  flown  (commanders,  operations 
and  intelligence  officers)  should  have  been  elicited  rather  than  the  judgment 
only  of  those  who  execute  the  missions. 


20.  ARE  GUESSES  AND  INTUITIVE  JUDGMENTS  IDENTIFIED? 

At  times  it  is  necessary  to  fill  in  data  gaps  with  educated  guesses  and 
intuitive  judgment.  These  educated  guesses  and  judgments  should  be 
identified  in  the  study  and  not  "  swept  under  the  rug.  "  The  reviewer 
should  evaluate  these  judgments  and  weigh  their  impact  on  the  conclusions 
and  recommendations. 
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COST 


21.  IS  THE  COST  MODEL  IDENTIFIED? 

Every  cost-effectiveness  analysis  contains  a  cost  model.  A  cost 
model  generates  cost  estimates  by  application  of  cost  estimating  relations 
and  cost  factors  to  specified  physical  resources.  (For  a  further  discus¬ 
sion  on  models  in  general  see  question  37.)  This  model  can  be  very 
complex  and  computer  assisted  or  it  may  consist  of  a  few  relatively  simple 
equations  readily  computed  by  hand.  The  study  should  sufficiently  identify 
the  cost  model  so  that  the  reviewer  can  determine  how  the  total  system 
cost  estimates  were  derived  from  the  material  in  the  study.  If  the  mater¬ 
ial  in  the  study  does  not  permit  the  reviewer  to  do  this,  then  additional 
information  is  required  from  the  agency  that  prepared  the  study. 

The  cost  models  are  utilized  to  estimate  the  probable  economic  im¬ 
pact  on  the  Service  (or  Nation)  of  introducing  a  new  capability.  For 
planning,  these  costs  are  normally  stated  in  terms  of  research  and  de¬ 
velopment  costs,  investment  costs,  and  operating  costs.  Research  and 
development  costs  include  those  costs  primarily  associated  with  the 
development  of  a  new  capability  to  the  point  where  it  is  ready  for  opera¬ 
tional  use.  Investment  costs  are  those  costs  beyond  the  development  phase 
to  introduce  a  new  capability  into  operational  use.  Operating  costs  are 
recurring  costs  required  to  operate  and  maintain  the  capability. 


22.  ARE  THE  COST  ESTIMATES  RELEVANT? 

Cost  estimates  depend  on  the  problem  under  study  and  can  rarely  be 
obtained  from  books  containing  cost  data  although  cost  factors  and  cost 
estimating  relations  (CERs)  can  sometimes  be  found  in  such  books.  For 
example,  a  hypothetical  study  considers  as  an  alternative  a  new  kind  of 
light  infantry  division  which  has  been  designed  to  the  extent  of  an  outline 
TOE.  The  answer  to  the.  seemingly  simple  question,  "  What  is  the  cost  of 
this  new  division?  "  depends  on  many  factors  including: 

Will  it  be  an  additional  division  to  those  already 
in  the  force  structure? 

Will  it  replace  an  existing  division?  If  so,  what 
kind? 

Where  will  it  be  stationed?  e.g.  ,  in  the  CONUS, 

Pacific,  Europe,  etc . 

Will  it  have  new  Standard  A  equipment,  or  will 
existing  assets  of  Standard  B  type  equipment  be 
used? 
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Are  there  any  existing  Army  units  whose  personnel, 
equipment,  and  facilities  can  be  used  by  the  new 
division? 

The  determination  of  which  costs  are  relevant  requires  considerable 
analysis  and  judgment.  It  is  not  possible  to  prepare  a  universal  list  of 
costs  that  are  always  relevant.  Ideally,  a  study  should  indicate  why  cer¬ 
tain  costs  were  considered  relevant  and  others  not.  The  questions  that 
follow  are  designed  to  help  the  reviewer  determine  whether  the  cost 
estimates  used  in  a  study  are  relevant. 


23.  ARE  INCREMENTAL  COSTS  CONSIDERED? 

Inherited  assets  are  those  resources  such  as  installations,  equipment, 
and  trained  personnel  inherited  from  earlier  systems  which  are  phasing 
out  of  the  force  structure  and  are  usable  in  one  or  more  of  the  alternatives 
under  study.  The  costs  which  are  usually  pertinent  for  planning  purposes 
are  those  costs  yet  to  be  incurred.  For  example,  a  study  considers  as  an 
alternative  the  conversion  of  certain  artillery  units  from  tube  to  missile 
weapons.  In  determining  the  incremental  costs  consideration  should  be 
given  to  the  inherited  assets  of  trained  personnel,  equipment,  and  facili¬ 
ties  that  are  or  can  readily  be  made  common  to  both  units. 

Sunk  costs  are  those  costs  already  expended.  These  previously  in¬ 
curred  costs  are  normally  excluded  from  costs  presented  in  cost-effective¬ 
ness  analysis.  For  example,  a  study  considers  as  possible  alternatives 
weapons  systems  A,  B,  and  C,  each  with  an  associated  research  and 
development  cost.  Only  alternative  A  is  already  under  development.  The 
cost  already  expended  on  Alternative  A  is  a  sunk  cost  and  the  research  and 
development  cost  of  Alternative  A  in  the  study  should  be  only  what  must 
yet  be  spent  (to  complete  the  research  and  development  of  Alternative  A). 

An  occasional  error  is  the  failure  to  consider  the  research,  develop¬ 
ment  and  investment  costs  of  existing  systems  as  sunk  costs.  For  example, 
in  a  hypothetical  study  of  the  conversion  of  certain  artillery  units  from  tube 
to  missile  weapons,  one  of  the  alternatives  is  retention  cf  all  of  the  tube 
weapons  units.  The  cost  of  that  alternative  would  not  include  the  sunk  costs 
represented  by  the  research  and  development  and  investment  costs  already 
expended  in  bringing  those  units  into  the  force  structure. 

Including  the  costs  of  inherited  assets  and  other  sunk  costs  leads  to 
distorted  cost  estimates  with  consequent  effect  on  the  conclusions  and 
recommendations. 
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24.  ARE  DIRECTLY  RELATED  SUPPORT  COSTS  INCLUDED? 

Cost  estimates  of  systems  or  organizations  should  include  the  pro¬ 
portionate.  cost  of  those  other  units  or  elements  required  in  direct  support. 
For  example,  the  cost  estimate  of  HAWK  battalions  should  include  the 
costs  of  the  associated  HAWK  direct  and  general  support  detachments.  In 
the  same  manner,  the  cost  of  aviation  units  should  include  a  direct' share 
of  the  cost  of  aviation  maintenance  units.  Failure  to  include  directly  re¬ 
lated  support  costs  may  result  in  misleading  cost  estimates  of  alternatives 


25.  ARE  COMBAT  CONSUMPTION,  REPLACEMENT/  CONSUMPTION, 

AND  MAINTENANCE  FLOAT  COSTS  INCLUDED? 

Cost  estimates  for  the  major  equipment  items  should  include  not  only 
the  operational  equipment  assigned  to  organizations,  but  also  the  costs 
for  those  additional  items  required  for  initial  stockage  as  well  as  replace¬ 
ment  items  over  the  period  in  which  the  system  is  to  be  in  use.  (See 
question  32.)  If  the  resource  implications  for  procuring  and  maintaining 
authorized  maintenance  float,  replacement/  consumption,  and  combat  con¬ 
sumption  stockage  are  excluded,  the  total  costs  of  the  system  alternatives 
may  be  significantly  misleading.  (These  levels  of  stockage  are,  of  course 
subject  to  logistics  guidance.)  For  example,  a  common  error  is  to  in¬ 
clude  only  the  cost  cf  the  basic  load  of  ammunition  and  to  neglect  the  cost 
of  the  additional  ammunition  requirements  for  support  of  the  weapon  sys¬ 
tem  or  organization.  The  total  ammunition  required,  to  include  peacetime 
training  requirements  and  expenditures  in  the  first  part  of  a  war  until 
wartime  production  becomes  available,  must  be  purchased  and  stocked  in 
peacetime. 


26.  ARE  ALL  TRAINING  COSTS  INCLUDED? 

The  resource  implications  of  training  military  personnel  can  be 
significant.  Initial  training  costs  represent  the  resources  required  for 
the  training  of  personnel  necessary  for  introduction  of  the  alternative  into 
the  force  structure.  The  availability  of  fully-trained  personnel,  as  well 
as  the  number  of  personnel  requiring  complete  or  transitional  training, 
are  taken  into  consideration  in  determining  the  resources  required. 

Annual  training  costs  represent  the  resource  implications  for  training  re¬ 
placements.  These  replacements  are  required  because  of  normal 
attrition. 

Training  costs  usually  include  such  items  as:  (1)  procurement  of 
equipment  utilized  for  training  purposes;  (2)  construction  of  any  neces¬ 
sary  additional  facilities;  (3)  operation  and  maintenance  costs  of  the 
facilities;  (4)  the  pay  and  allowances  of  the  trainees.  For  example,  the 
cost  implications  of  communications-electronic  equipment  utilized  for 
training  purposes  could  be  highly  significant. 
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27.  ARE  CONSTRUCTION  COSTS  INCLUDED? 

The  costs  for  additional  installations  or  facilities  are  often  overlooked 
yet  these  costs  can  be  important.  If  the  study  does  not  include  any  con¬ 
struction  costs  and  does  not  state  how  the  facilities  were  obtained,  then 
the  reviewer  must  either  satisfy  himself  that  no  construction  is  required 
or  take  necessary  steps  to  have  the  study  corrected. 


28.  ARE  THE  COST  DATA  REASONABLY  ACCURATE? 

Although  it  is  not  usually  possible  for  a  reviewer  to  check  all  cost 
data  for  accuracy,  he  should  spot-check  and  examine  the  sources  of  the 
data . 

Cost  data  furnished  by  manufacturers  should,  be  viewed  critically. 
Experience  has  shown  that  such  data  are  usually  understated,  particularly 
for  advanced  systems.  Advanced  system  costs  stated  as  an  exact  figure 
rather  than  as  estimated  lower  and  upper  values  are  particularly  suspect. 

The  basis  of  the  cost  data  for  advanced  systems  should  be  included  in 
the  study.  There  are  a  number  of  ways  for  arriving  at  such  estimates. 
One  commonly  accepted  method  relates  the  cost  data  for  the  components 
of  existing  analogous  systems  to  the  cost  of  the  advanced  system.  Unsup¬ 
ported  cost  data  are  suspect. 

Great  accuracy  in  cost  estimates  is  not  required  and  often  is  not 
feasible.  In  fact,  in  dealing  with  costs  of  advanced  systems  it  is  usually 
more  realistic  to  have  a  range  of  possible  costs  (upper  and  lower  values) 
rather  than  the  pseudo-accuracy  of  one  cost  figure  which  assumes  no 
uncertainties  in  arriving  at  that  figure. 


29.  ARE  COST  ASPECTS  OF  ALL  ALTERNATIVES  TREATED  IN  A 

COMPARABLE  MANNER? 

Inconsistency  in  handling  the  cost  aspects  cf  competing  alternatives 
prevents  an  objective  evaluation  of  their  comparative  or  relative  costs 
and  usually  leads  to  erroneous  conclusions.  It  is  not  always  possible  to 
use  the  same  cost  estimating  technique  for  calculating  a  cost  element  such 
as  attrition  replacements.  This  is  often  the  case  in  studies  involving 
alternative  systems  of  other  military  services.  For  example,  one  service 
may  calculate  aircraft  attrition  replacement  as  a  function  of  an  activity 
rate  (e.g. ,  per  100,000  flying  hours)  while  another  service  may  calculate 
it  as  a  function  of  the  activity  inventory  (3  percent  of  the  active  inventory 
per  year).  The  reviewer  should  determine  that  the  final  dollar  estimate 
is  related  to  the  actual  resource  requirements  for  the  alternative  and  that 
computational  peculiarities  do  not  distort  the  cost  results. 
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Treating  alternatives  in  a  comparable  manner  must  not  be  carried  to 
the  point  that  costs  which  may  be  insignificant  in  one  alternative  are 
therefore  not  considered  at  all  in  other  alternatives.  For  example, 
civilian  personnel  might  not  be  used  in  one  alternative  but  may  be  required 
by  another  alternative  in  significant  numbers.  To  exclude  this  cost  could 
distort  the  results. 


30.  ARE  THE  COST  ESTIMATING  RELATIONS  VALID? 

Cost  estimating  relations  may  be  crude  factors,  simple  ex-zapolation 
of  recent  experience,  or  complex  equations  with  many  variables.  In  all 
cases,  the  purpose  of  a  cost  estimating  relation  is  to  translate  a  specifi¬ 
cation  of  a  physical  resource  into  a  cost.  The  design  of  valid  cost 
estimating  relations  is  a  complex  subject  beyond  the  scope  of  this  publi¬ 
cation.  However  ,  several  common  errors  made  in  establishing  cost 
relations  are  discussed  below. 

Cost  estimating  relations  should  be  based  on  current  data  or  distorted 
estimates  may  result.  For  example,  the  maintenance  cost  per  flying  hour 
for  an  Army  helicopter  has  decreased  significantly  over  the  past  several 
years  as  new  helicopters  have  been  introduced  into  the  force  structure.  If 
the  cost  estimating  relations  used  in  a  study  were  based  on  information  for 
early  Army  helicopters  (e.  g. ,  1946  through  1954  data)  the  maintenance 
cost  per  flying  hour  for  a  present  system  as  well  as  for  future  systems 
alternatives  could  be  distorted. 

At  times  a  properly  constructed  cost  estimating  relation  may  be 
inapplicable.  If  the  system  alternatives  are  very  advanced  developments, 
the  cost  estimating  relations  based  on  the  current  technology  may  lead  to 
false  results.  For  example,  the  V/ STOL  aircraft  concept  represents  a 
departure  from  aircraft  currently  in  production.  While  many  design 
characteristics  may  be  similar  to  present  aircraft,  there  may  be  a  number 
of  factors  which  could  increase  the  complexity  and  hence,  the  cost  of  the 
aircraft;  a  cost  estimating  relation  based  on  the  present  state-of-the-art 
may  not  be  appropriate. 


31.  IS  AN  AMORTIZED  COST  USED? 

Amortized  costs  reduce  the  total  program  cost  cf  the  system  to  an 
annual  cost  by  taking  the  total  operating  cost  cf  the  program,  adding  to  it 
the  research  and  development  and  investment  costs,  and  reducing  the 
total  to  an  annual  basis  by  dividing  by  the  number  of  years  of  expected 
service  life  of  the  system.  The  same  general  procedure  may  be  utilized 
to  amortize  the  costs  per  month,  per  day,  per  sortie,  etc.  This  approach 
disguises  the  differences  between  annual  operating  costs  resulting  from 
shifting  deployment  patterns  over  the  life  of  the  system  and  from  a  varying 
set  of  inherited  assets  over  time.  This  approach  makes  an  arbitrary 
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allocation  of  the  fixed  costs  of  a  system  over  time.  There  is  no  basis  for 
the  assumption  that  the  last  year  of  system  life  must  be  charged  with  the 
same  amount  of  R&D  cost  as  the  first  year.  The  first  year  gets  the  newest 
technology;  the  last  year,  obsolete  technology.  Further,  an  amortized  cost 
does  not  present  a  true  picture  of  the  total  resource  implications.  If  the 
system  is  to  be  in  the  force  structure  for  say  10  years,  the  amortized 
annual  costs  may  look  relatively  small,  yet  in  reality  there  may  be  rela¬ 
tively  large  dollar  costs.  It  is  the  total  cost  of  the  alternatives  which  is  of 
primary  concern. 

The  reviewer  should  attempt  to  convert  amortized  costs  into  total 
program  costs  and  use  such  costs  €or  comparative  purposes.  If  this  can¬ 
not  be  done  readily  from  the  material  contained  in  the  study,  then  additional 
information  is  required  from  the  study  agency. 


32.  WERE  PEACETIME  OR  WARTIME  COSTS  INCLUDED? 

The  results  of  a  cost-effectiveness  study  may  be  very  sensitive  to  the 
use  made  of  peacetime  and  wartime  costs.  The  use  cf  peacetime  costs 
only  may  indicate  that  System  A  is  preferred  while  the  same  study,  if  war¬ 
time  costs  were  used,  may  have  concluded  that  System  B  is  preferable. 

Peacetime  costs  may  be  defined  as  the  costs  associated  with  develop¬ 
ing,  buying,  and  maintaining  a  capability  for  potential  war  during  peacetime. 
Such  costs  also  include  combat  consumption  stocks  (war  reserves)  to  cover 
the  period  from  the  beginning  of  a  war  until  wartime  production  is  able  to 
replace  battle  losses.  Wartime  costs  are  the  costs  of  procurement  after 
the  war  has  begun,  as  is  the  cost  of  replacing  the  combat  consumption 
stocks  if  the  war  terminated  during  the  useful  life  cf  the  system. 

In  the  case  of  general  purpose  forces  there  may  be  significant  pro¬ 
duction  of  weapons  and  expenditure  of  resources  after  a  limited  conflict 
begins  (as  in  the  Korean  Conflict  and  the  military  assistance  rendered  to 
South  Vietnam).  In  this  case,  wartime  costs  could  be  significant.  How¬ 
ever,  wartime  costs  are  difficult  to  determine  because  of  uncertainty 
regarding  the  duration  of  the  war,  loss  rates;  and  missions  undertaken. 

The  reviewer  should  be  guided  in  considering  the  proper  cost  approach 
(peacetime  or  wartime)  by  existing  policy  or  directive  from  the  agency 
directing  that  the  study  be  made. 


33.  WAS  A  WARTIME  ORDNANCE  COST  PER  MISSION  USED? 

The  use  of  a  wartime  ordnance  cost  per  mission  should  be  reviewed 
carefully.  Variations  on  this  approach  include  ordnance  cost  per  target 
killed,  per  casualty  and  per  sortie.  This  approach  is  usually  deficient 
because  of  failure  to  consider  all  the  costs  of  putting  into  place  and 
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maintaining  a  capability  for  potential  war  throughout  the  projected  life  of 
the  system  in  the  active  force  structure.  Often,  this  approach  includes 
only  the  ammunition  costs  expended  during  a  brief  battle,  and  neglects  the 
bulk  of  the  significant  costs  associated  with  developing,  buying  and  opera¬ 
ting  the  system  in  peacetime. 


34.  WAS  AN  AMORTIZED  WARTIME-PEACETIME  COST  USED? 

In  this  approach  the  total  peacetime  cost  of  the  system  is  reduced  to 
an  annual  basis  as  explained  in  question  31.  To  this  amortized  peacetime 
cost  is  added  the  estimated  annual  wartime  replacement/  consumption 
costs.  No  distinction  is  made  between  wartime  and  peacetime  costs.  This 
approach  is  deficient  because:  (1)  it  assumes  the  war  will  continue  over 
the  entire  projected  service  life  (f  the  system;  (2)  the  cost  results  use 
weighted  wartime  costs;  (3)  wartime  and  peacetime  costs  may  not  be 
commensurable;  and  (4)  it  does  not  present  a  true  picture  cf  the  total  re¬ 
source  implications  as  discussed  in  question  31  . 

Adding  amortized  costs  in  one  stream  to  another  annual  cost  stream 
infers  that  both  cost  streams  represent  the  same  total  time  duration.  If 
this  is  not  the  case,  then  the  two  cost  streams  should  not  be  added  together 
because  they  are  incommensurables.  Adding  the  amortized  peacetime 
costs  to  the  annual  wartime  cost  implicitly  assumes  that  the  war  will  con¬ 
tinue  over  the  entire  "  service  life"  of  the  system.  If  the  peacetime  costs 
had  been  amortized  to  a  per  day  or  per  mission  cost  instead  of  a  per  year 
basis  ,  the  same  result  would-  hold,  the  inference  being  that  the  war  would 
continue  over  the  entire  "  service  life"  of  the  system.  The  implied  assump¬ 
tion  that  the  war  would  last  for  the  "  service  life"  of  the  weapon  system  is 
questionable. 

Costs  computed  by  this  method  are  weighted  because  wartime  costs 
do  not  cover  the  same  length  of  time  as  peacetime  costs.  Such  weighted 
results  favor  the  shorter  time  period--the  wartime  costs.  It  is  only  when 
the  two  cost  streams  are  of  equal  length  that  the  costs  results  are  not 
distorted. 

To  assume  that  wartime  and  peacetime  costs  are  commensurable  may 
be  erroneous.  This  assumes  a  common  measure  between  the  values  of 
resources  procured  in  wartime  and  those  procured  in  peacetime.  During 
wartime,  the  cost  of  a  resource  may  be  quite  different  from  that  in  peace¬ 
time.  Military  budget  constraints  during  peacetime  and  physical  resource 
constraints  during  wartime  may  produce  entirely  different  sets  of  costs  for 
the  same  military  resources.  Asa  greater  proportion  cf  the  national  bud¬ 
get  is  shifted  to  military  purposes  during  wartime,  the  scarcity  cf  dollars 
for  military  resources  may  become  relatively  less  or  more  than  during 
peacetime.  Commensurability  between  wartime  and  peacetime  costs  will 
depend  upon  such  uncertainties  as  the  type  and  duration  of  war  and  whether 
a  war  will  actually  occur. 
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35.  WAS  A  DOLLAR  COST  ASSIGNED  TO  THE  LOSS  OF  HUMAN  LIFE? 

Frequently,  a  study  will  assign  a  dollar  cost  to  a  human  casualty.  The 
loss  cf  human  life  is  certainly  important  in  selection  among  alternatives. 
However,  the  value  of  a  human  life  is  incommensurable  with  the  dollar 
costs  associated  with  an  alternative.  It  is  better  to  treat  human  losses  as 
a  separate  measure  without  assigning  dollar  values.  Manpower  availability 
in  both  peace  and  war  is  very  important  but  this  problem  cannot  be  properly 
treated  only  in  terms  of  dollar  costs.  Men  and  dollars  may  not  be  inter¬ 
changed. 


36.  IS  THE  SENSITIVITY  OF  COST  ASSUMPTIONS  EXAMINED? 

In  comparing  costs  cf  alternative  systems,  it  is  important  to  determine 
whether  the  results  are  independent  of  the  cost  assumptions.  For  example, 
would  ten  years  of  peacetime  operations  as  opposed  to  five  make  a  signifi¬ 
cant  difference  in  the  relative  costs  of  the  alternatives?  Would  it  make 
any  difference  if  the  procurement  levels  or  number  of  units  to  be  organized 
changed?  The  study  should  make  clear  the  sensitivity  of  the  cost  estimates 
to  the  major  cost  assumptions.  If  the  study  fails  to  do  this,  the  reviewer 
should  attempt  to  determine  if  there  is  any  such  significant  sensitivity  by 
rough  calculation. 
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RELATIONSHIPS  (MODELS) 


37.  ARE  THE  MODELS  ADEQUATELY  IDENTIFIED  AND  EXPLAINED? 

The  conclusions  and  recommendations  of  a  cost-effectiveness, 
analysis  cannot  be  evaluated  properly  unless  the  models  are  adequately 
identified  and  explained.  Every  model  portrays  the  real  or  expected 
world  by  abstraction  and  simplification  in  order  to  predict  the  outcome 
of  a  possible  action  (see  Glossary),  Therefore,  the  explanation  of  the 
model  should  be  sufficient  to  provide  ready  understanding  of  which  as¬ 
pects  of  the  real  world  are  included  in  the  model,  which  aspects  have 
been  omitted,  and  the  underlying  assumptions  for  the  abstraction.  Basic¬ 
ally,  a  good  model  emphasizes  the  specific  areas  in  which  decisions 
are  to  be  made  by  removing  those  relatively  constant  elements  of  the 
real  or  expected  world  that  can  be  described  with  a  great  degree  of 
certainty . 

The  study  should  contain  sufficient  explanation  to  permit  tracing 
the  operation  of  the  model  from  input  to  output.  The  detail  should  be 
sufficient  to  permit  calculation  of  new  results  from  different  input  values 
(sensitivity  analysis).  In  cases  where  a  model  is  machine-programmed, 
sufficient  explanation  should  be  provided  for  following  the  general  logic 
of  the  program. 

38.  ARE  COST  AND  EFFECTIVENESS  LINKED  LOGICALLY? 

A  properly  structured  cost-effectiveness  analysis  contains  a  number 
of  models  that  link  effectiveness  and  cost  through  logical  interrelations. 
Usually  there  are  some  kinds  of  an  effectiveness  model,  a  system  and 
organization  model,  a  cost  model,  and  a  cost-effectiveness  model.  The 
exact  nature  and  number  of  these  models  will  vary  with  the  problem. 

The  study  should  provide  sufficient  information  and  explanation  for  the 
reviewer  to  follow  the  logic  by  which  the  models  relate  cost  and  effect¬ 
iveness. 

An  effectiveness  model  relates  measures  of  effectiveness  to  measures 
of  performance  in  an  operational  context.  For  example,  a  study  on  com¬ 
bat  vehicle  weapons  systems  used  as  a  measure  of  effectiveness  the 
probability  of  1,2,  3,  ...  friendly  tanks  winning  an  engagement  with 
1,2,3,  ...  enemy  tanks  under  different  tactical  situations.  This  was 
related  to  performance  measures  such  as  muzzle  velocity  ,  warhead 
specifics,  turret  slewrates,  turret  stability,  hull  characteristics,  rate 
of  fire,  target  acquisition  accuracy,  and  others,  under  various  tactical 
situations  and  rules  for  conduct  of  fire. 

A  system  and  organization  model  describes  the  physical  resources 
required  to  provide  the  performance  used  in  the  effectiveness  model. 

For  example,  in  the  combat  vehicle  weapons  system  study  referred  to. 
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this  included  the  physical  description  of  each  alternative,  the  complete 
vehicle,  ammunition,  armament,  fire  control,  communications,  TOE 
unit  description,  the  support  and  maintenance  requirements,  and  so 
forth,  consistent  with  the  planned  operational  concept. 

A  cost  model  relates  dollar  costs  to  the  physical  resources  (and 
their  peacetime  activity  rates)  described  in  the  system/organization 
model.  The  cost  model  applies  cost  estimating  relations  and  factors. 

For  example,  the  same  study  used  the  total  future  cost  of  aquisition  and 
ownership  (R&D,  initial  investment,  annual  operating)  for  various  quan¬ 
tities  of  systems.  Included  in  these  total  costs  were  not  only  the  develop¬ 
ment  and  procurement  of  the  preferred  item  but  also  such  additional 
costs  caused  by  training  of  personnel,  peacetime  ammunition  use,  equip¬ 
ment  maintenance,  etc.  (See  question  22). 

The  cost-effectiveness  model  finally  relates  the  costs  of  each  al¬ 
ternative  to  its  effectiveness  under  varying  assumptions.  Depending  on 
the  criterion,  the  model  may  compare  effectiveness  and  costs  of  alter¬ 
natives  at  equal  cost,  at  equal  effectiveness,  or  at  different  cost  and 
different  effectiveness  (seepage  9).  The  method  and  the  techniques 
used  to  achieve  this  cost  and  effectiveness  relation  should  be  logical  and 
explained.  For  example,  in  one  anti-tank  weapons  study  the  equal 
effectiveness  method  (winning  the  duel  -  all  pertinent  factors  considered) 
was  employed.  Effectiveness  was  related  to  cost  by  a  numerical  formula 
for  calculation  of  cost  of  achieving  duel  success  at  a  given  range  under 
specified  conditions.  This  permitted  plotting  the  following  graph: 


$  20,000 


Range 

Figure  1.  Equal  Effectiveness  Method 
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The  graph  shows  the  cost  of  winning  a  "duel",  i.  e.  ,  killing  the 
target  at  various  ranges.  (The  graph  portrayed  above  is  highly  simpli¬ 
fied.  In  the  actual  study  rather  than  a  simple  line,  a  band  was  used  to 
portray  the  variance  in  costs  for  winning  a  duel  at  a  given  range.  See 
Bibliography  Item  No.  1,  pages  13- 17  for  more  complete  description). 

39.  DOES  THE  MODEL  TREAT  THE  PROBLEM  IN  A  SYSTEM 

CONTEXT? 

Most  military  systems  have  many  subsystems,  sub-subsystems  and 
so  forth.  Models  should  provide  for  the -proper  relations  among  subsys¬ 
tems  so  that  the  full  implications  of  a  change  in  one  part  of  the  system 
will  be  reflected  in  the  rest  of  the  system.  For  example,  a  model  in  a 
study  of  an  airborne  surveillance  system  must  not  only  show  the  inter¬ 
relations  among  the  aircraft  (or  drones),  the  sensors  and  their  main¬ 
tenance,  but  also  the  interrelations  with  the  information  processing 
functions  to  be  performed  on  the  ground. 

40.  DOES  THE  MODEL  ALLOW  FOR  ENEMY  REACTION? 

It  normally  takes  several  years  to  implement  fully  a  decision  to 
deploy  a  new  system.  Therefore,  the  enemy  should  be  considered  to 
have  time  to  adjust  to  our  system  decisions.  A  major  aspect  of  the 
effectiveness  of  our  system  is  the  degree  to  which  it  makes  such  adapt¬ 
ation  for  the  enemy  either  technologically  difficult  or  economically 
unattractive . 

For  example,  a  study  of  a  proposed  system  was  based  on  its  incor¬ 
poration  into  the  current  force  structure.  The  model  forjudging  the 
effectiveness  of  the  proposed  system  was  dominated  by  current  or 
recent  conflict  situations  (e.g.  ,  Vietnam,  Korea,  Europe).  In  using  the 
model  to  evaluate  the  effectiveness  of  the  future  system  only  in  the  light 
of  these  current  or  recent  conflict  scenarios,  the  study  failed  to  con¬ 
sider  the  steps  that  the  enemy  could  take  to  counter  the  proposed  system. 
(See  question  6). 

41.  ARE  STRAIGHT  EXTRAPOLATIONS  USED  WITHOUT  PROOF? 

While  straight  extrapolations  (linear  relation)  often  do  apply  over 
limited  ranges  of  performance,  consumption,  or  similar  planning  figures 
based  on  averages  of  large  numbers,  they  rarely  apply  to  effectiveness 
o  r  cost  data. 

For  example,  the  relation  between  the  total  weight  of  rations  for 
one  infantry  division- month  and  the  weight  for  10  division-months  is  a 
straight  extrapolation.  The  relation  between  the  total  cost  of  the  first 
100  and  that  of  the  first  1,  000  units  of  a  new  main  battle  tank  is  not 
linear  or  a  straight  extrapolation.  If  a  missile  system  has  10  missiles, 
costs  $1,000,000,  and  is  50%  effective  (on  some  valid  measure),  then 
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a  missile  system  with  20  missiles,  costing  $2,000,000,  will  not  be  100% 
effective  but  (at  best)  75%. 

42.  ARE  DETERMINISTIC  AND  PROBABILISTIC  MODELS  USED 

PROPERLY? 

A  deterministic  model  (see  Glossary)  uses  relations  of  the  type, 

"If  A  is  5,  then  B  is  always  8".  A  probabilistic  model  (see  Glossary) 
uses  relations  of  the  type,  "If  A  is  5,  then  B  will  be  6-10  in  50%of  the 
cases,  4  or  5  in  25% of  the  cases,  and  11  or  12  in  25%  of  the  cases". 

Cost-effectiveness  analyses  frequently  require  many  intermediate 
calculations  involving  data.  The  indiscriminate  use  of  specific  values 
often  creates  what  is  in  effect  a  deterministic  model.  In  reality,  the 
majority  of  the  coefficients  and  planning  factors  used  in  models  are 
only  averages  with  variances  and  different  degrees  of  confidence.  The 
reviewer  should  try  to  identify  the  probable  range  of  variance  about  the 
averages  that  are  used  as  inputs  and  have  at  least  an  intuitive  feeling 
about  the  confidence  of  the  numerical  results. 

Additionally  ,  the  reviewer  should  distinguish  those  cases  in  which 
a  probabilistic  model  is  needed  to  reflect  the  real  world  situation. 
Deterministic  models  are  usually  appropriate  (l)when  the  planning  fac¬ 
tor  has  an  insignificant  variance,  such  as  weight  of  rations  per  day  per 
man  for  large  forces,  (2)  if  the  uncertain  factor  is  multiplied  by  a  point 
value,  such  as  cost  of  $8,000  to  $12,000  per  man  for  a  force  of  20,000 
men,  (3)  a  varying  factor  is  multiplied  by  a  linear  function,  such  as  an 
uncertain  flying  hour  rate  (e.  g.  ,  2  to  6  hours  per  day)  multiplied  by  a 
flying  hour  cost  function  of  $20  a  day  plus  $40  per  flying  hour.  The  deter¬ 
ministic  technique  is  correct  in  these  three  cases  because  it  will  give 
the  same  most  probable  result  as  if  probabilistic  techniques  had  been 
applied.  Of  course,  there  may  still  be  a  problem  if  the  most  probable 
result  is  not  the  only  one  of  interest. 

Probabilistic  models  are  used  where  the  variables  in  the  problem 
may  assume,  at  any  given  time,  any  one  value  of  a  known  range  and 
frequency  of  values,  as  opposed  to  deterministic  models  which  use 
fixed  or  average  values  all  the  time.  There  are  two  principal  types  of 
probabilistic  models:  static  models  using  probability  statements  instead 
of  other  values,  and  dynamic  (stochastic)  models  involving  change. 

Some  stochastic  models  use  random  numbers,  representing  change, 
to  select  values  from  frequency  distributions  for  a  given  problem.  For 
example ,  an  analysis  cf  a  maintenance  support  organization  may  include 
a  model  which  represents  the  demands  for  maintenance  feffort  placed  on 
the  support  organization.  Of  any  lOO  jobs  (demands),  20  will  require 
1  man-hour, -30  will  require  2  manhours,  10  will  require  3  manhours, 

15  will  require  4  manhours,  5  will  require  5  manhours,  10  will  require 
10  manhours,  5  will  require  20  manhours,  2  will  require  30  manhours. 
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2  will  require  40  manhours,  and  Iwill  require  80  manhours.  This 
information  is  arranged  into  a  cumulative  distribution  as  shown  below: 
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To  represent  requests  for  work,  a  two-digit  random  number,  say, 

37,  is  drawn  (from  a  table  of  random  numbers  or  a  random  number 
generator);  the  corresponding  value  is  2  manhours.  The  next  random 
number  is,  say  84,  and  the  value  is  10  manhours.  This  process  con¬ 
tinues  at  some  rate  (which  is  probabilistic)  and  the  requests  for  main¬ 
tenance  are  arranged  (queued)  in  the  order  of  simulated  requests:  2 
manhours,  10  manhours,  4  manhours,  and  so  forth.  Available  mainte¬ 
nance  men  would  be  assigned  to  requests  under  various  rules,  e.  g.  , 
lman  to  jobs  less  than  4  hours,  2  men  to  jobs  of  4  to  8  hours,  etc.  The 
model  would  keep  track  of  the  time  elapsed  between  generating  and 
completing  a  request  for  maintenance.  In  this  manner,  the  relation  of 
number  of  maintenance  personnel  and  delay  can  be  determined  for  various 
assumptions  about  demand  for  maintenance  effort. 

The  so-called  Monte-Carlo  model  described  above  requires,  however, 
a  sufficient  number  of  repetitions  to  obtain  adequate  information  about 
the  range  of  values  of  the  solution. 

A  static  model  using  probability  statements  may,  for  example, 
apply  in  a  study  on  aircraft  vulnerability.  The  probability  of  survival 
for  a  specified  time  is  given  by  the  product  resulting  from  the  multipli¬ 
cation  of  the  following  probabilities: 

Probability  of  aircraft  being  detected 
Probability  of  aircraft  being  acquired  by  a  weapon, 
if  detected 

Probability  of  being  hit,  if  acquired  by  a  weapon 
Probability  of  kill,  if  hit. 

Probability  data  for  each  of  the  probabilities  listed  above  are  de¬ 
rived  from  tests  and  experiments. 

43.  IS  A  ZERO-SUM  GAME  MODEL  USED  WHERE  IT  IS  NOT 

APPLICABLE  ? 

A  zero-sum  two-person  game  is  a  conflict  in  which  there  are  two 
sides  and  the  gains  of  one  side  equal  the  losses  of  the  other.  Most  con¬ 
flict  situations  do  not  justify  the  use  of  this  type  of  model.  For  example, 
in  a  hypothetical  study,  the  effectiveness  of  alternative  US  tank  systems 
was  based  on  a  study  of  duels  between  US  tanks  and  enemy  tanks.  Duel 


D-34 


AMCP  706-191 


results  were  based  on  the  losses  incurred  by  each  side.  An  enemy  loss 
of  one  tank  was  equated  to  a  US  gain  of  one  tank.  The  net  US  gain  was 
used  to  determine  the  effectiveness  for  each  alternative. 

Our  gain  is  not  the  enemy's  loss.  The  situation  is  not  always 
symmetrical.  The  attacker  must  move,  the  defender  must  inhibit  move¬ 
ment.  Hence,  the  objective  of  a  US  tank  may  differ  from  the  objective 
of  the  enemy  tank.  In  fact,  other  alternative  concepts  might  inhibit 
enemy  tank  movement  more  effectively  than  would  a  US  tank  similar  to 
an  enemy  tank. 

44.  ARE  THE  MODELS  INTUITIVELY  ACCEPTABLE? 

Models  tend  to  become  mathematical  and  many  are  difficult  to  under¬ 
stand  even  in  their  broad  aspects.  Yet,  overly-simplified  models  tend 
to  become  superficial  by  limitation  in  choice  of  detail  and  omission  of 
important  variables.  The  objective  of  a  good  model  is  to  be  near  enough 
to  reality  so  that  the  model  outputs  can  be  used  to  predict  some  portions 
cf  the  future  with  an  acceptable  degree  cf  confidence. 

Models  can  be  tested  by  determining  if  they  represent  correctly 
known  facts  and  situations  not  considered  in  the  study.  Conversely,  if 
absurd  facts  and  situations  are  introduced  into  the  model,  comparable 
absurd  answers  should  be  produced  by  the  model.  If  the  reviewer  is 
aware  of  special  cases  in  which  there  is  some  indication  of  the  outcome, 
the  model  can  be  tested  to  determine  if  the  results  are  in  general  agree¬ 
ment  with  the  indicated  outcome.  Another  test  that  can  be  applied,  at 
times,  is  to  vary  some  of  the  principal  parameters  and  determine  if  the 
model  produces  results  that  are  consistent  and  plausible. 
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EFFECTIVENESS 

45.  ARE  THE  MEASURES  OF  EFFECTIVENESS  IDENTIFIED? 

The  study  should  clearly  identify  the  standards  or  measures  used 
for  evaluating  the  effectiveness  of  the  system  or  organization  under  study. 
If  not  explicitly  stated,  the  reviewer  should  attempt  to  identify  these 
measures  from  the  material  contained  in  the  study.  The  conclusions 
and  recommendations  cannot  be  properly  evaluated,  particularly  when 
the  study  is  based  on  equal  cost  alternatives,  without  prior  evaluation 
of  the  measures  of  effectiveness. 

46.  IS  THE  EFFECTIVENESS  MEASURE  APPROPRIATE  TO  THE 

FUNCTION  OR  MISSION? 

The  reviewer  should  satisfy  himself  that  the  measures  used  to  eval¬ 
uate  effectiveness  are  appropriate  to  the  function  or  mission  of  the 
system  or  organization  under  study.  Failure  to  use  meaningful  measures 
of  effectiveness  is  a  major  contributing  factor  to  unacceptable  studies. 
Examination  of  the  effectiveness  measures  requires  analysis  and  sound 
military  judgment.  The  example  below  illustrates  one  use  of  an  effec¬ 
tiveness  measure  that  was  not  appropriate. 

In  a  study  of  selected  infantry  and  artillery  weapons  systems,  the 
measure  of  effectiveness  was  a  division  firepower  score.  This  score 
was  the  sum  of  the  firepower  scores  of  the  units  within  the  division.  The 
firepower  score  of  a  unit  was  based  on  sustained  rates  of  fire,  effective 
width  of  burst,  and  the  fragmentation  area  of  the  weapon  in  comparison 
with  other  weapons.  Specifically,  direct- fire  weapons  such  as  rifles 
were  assessed  in  terms  of  probable  hits  per  minute  against  personnel  in 
the  open.  Mortars  and  artillery  were  assessed  in  terms  of  maximum 
effective  range  and  lethal  area  coverage  per  minute. 

This  use  of  a  firepower  score  was  wrong  for  a  number  of  reasons. 
Primarily,  it  failed  to  differentiate  between  the  effectiveness  of  weapons 
when  used  for  neutralization  and  when  used  to  produce  casualties.  For 
neutralization,  the  effectiveness  is  strongly  dependent  on  burst  rate  of 
fire,  incipient  damage  area  produced  by  the  burst,  and  ability  to  main¬ 
tain  fire  over  the  required  time  (the  latter  a  function  of  weapon  character¬ 
istics  and  ammunition  requirements).  On  the  other  hand,  casualty  pro¬ 
duction  depends  strongly  on  the  probability  of  hit,  which  in  turn  depends 
on  target  acquisition  and  weapon  guidance  or  accuracy.  Thus,  in  this 
case,  several  measures  must  be  used  to  have  a  valid  analysis. 

The  total  division  firepower  score  used  in  the  study  also  assumed 
an  inexhaustible  and  uniform  supply  of  ammunition  regardless  of  whether 
the  weapon  was  a  rifle  company  machine  gun  or  a  division  general  support 
artillery  cannon. 
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47.  DO  THE  EFFECTIVENESS  MEASURES  IGNORE  SOME  OBJECTIVES 

AND  CONCENTRATE  ATTENTION  ON  A  SINGLE  ONE? 

In  the  measurement  of  effectiveness,  the  reviewer  should  watch  for 
any  tendencies  to  concentrate  on  only  one  or  two  objectives.  Such  a 
situation  indicates  an  unstated  assumption  that  other  objectives  are 
unimportant.  The  resulting  conclusions  and  recommendations,  if  imple¬ 
mented,  may  cause  an  imbalance  and  reduce  the  capability  to  achieve 
other  objectives. 

For  example,  a  study  indicates  that  the  most  vulnerable  element  in 
a  line  of  communications  system  are  the  bridges  in  a  rail  network  and 
measures  effectiveness  of  deployment  of  given  air  defense  units  by 
degree  of  protection  afforded  railway' bridges.  In  evaluating  the  overall 
effectiveness  of  the  air  defense  deployment,  the  study  failes  to  consider 
that  the  vulnerability  of  other  elements  in  the  Tine  of  communications 
system  may  be  greatly  increased  by  the  redeployment  of  the  air  defense. 

A  possible  test  for  effectiveness  measures  suspected  of  concentrating 
on  a  single  objective  is  to  evaluate  them  against  a  hypothetical  obviously 
absurd  weapon  or  device  that  does  o.nly  one  job.  Valid  measures  of 
effectiveness  should  show  an  absurd  hypothetical  weapon  or  device  in  its 
true  light. 

48.  ARE  PERFORMANCE  MEASURES  MISTAKEN  FOR  EFFECTIVENESS 

MEASURES? 

Measures  of  performance  characteristics  are  sometimes  miscon- 
shed  as  measures  of  the  ability  of  the  system  or  organization  to  ac¬ 
complish  its  function.  Performance  characteristics  may  contribute 
one  of  the  many  inputs  required  to  achieve  the  effectiveness  of  the  system 
or  organization  as  a  whole.  For  example,  the  speed  of  movement  or 
mobility  of  a  unit  is  only  one  aspect  of  the  unit’ s  capability  to  accomplish 
its  function.  The  speed  at  which  a  unit  can  Attack  the  enemy  is  not  in  it¬ 
self  a  measure  of  the  ability  of  the  unit  to  defeat  the  enemy.  The  weapon 
with  the  smaller  CEP  is  not  necessarily  the  more  effective  weapon;  the 
relation  of  lethal  radius  to  CEP  may  be  more  significant.  Other  factors 
that  must  be  also  considered  in  weapon  effectiveness  include  target  ac¬ 
quisition,  weapon  guidance,  and  target  size. 

49.  IS  THE  EFFECTIVENESS  CALCULATED  ON  THE  BASIS  OF  EITHER 

A  COOPERATIVE  ENEMY  OR  AN  OMNIPOTENT  ENEMY? 

Neither  basis  is  valid.  The  enemy  should  be  expected  to  adjust 
his  decisions  to  our  own  planning  as  much  as  his  resources  permit.  An 
unstated  assumption  that  the  enemy  is  inflexible  in  the  face  of  our  changes 
is  a  common  error  in  cost-effectiveness  studies. 

For  example,  a  counter-guerrilla  study  used  a  scenario  in  which  the 
hostile  guerrilla  forces  retreat  to  a  mountain  redoubt  to  be  surrounded 
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by  US  troops  air-landed  by  helicopters.  This  scenario  makes  convention¬ 
al  tactics  palatable  in  counter-guerrilla  warfare,  but  is  hardly  realistic. 

A  capable  guerrilla  leader  should  not  be  expected  to  use  such  disastrous 
tactics.  Adaptation  of  enemy  tactics  (e.g.  ,  rapid  dispersal)  in  face  of 
the  new  US  capability  for  air  landing  is  certainly  feasible.  A  comparable 
adaptation  to  the  enemy  capabilities  was  illustrated  during  World  W  ar  11. 
German  air  defense  analyses  prior  to  that  war  were  based  on  the  attack¬ 
ing  aircraft  using  certain  altitudes  that  were  optimum  for  the  air  defense 
batteries.  Allied  bomber  aircraft  did  not  oblige  and  avoided  the  "optimum" 
altitude  range. 

Some  studies  assume  maximum  future  enemy  capability  in  all  weapon 
areas.  The  enemy  cannot  simultaneously  maximize  all  of  this  capabili¬ 
ties  if  constraints  in  physical  resources  and  budgets  are  present,  parti¬ 
cularly  in  the  case  of  peacetime  budgets.  If  he  maximizes  his  strategic 
forces,  he  will  have  to  limit  his  tactical  capabilities,  and  vice  versa. 
Alternatives,  where  appropriate,  should  be  pitted  against  a  variety  cf 
enemy  postures  and  the  choice  should  make  none  of  these  postures  parti¬ 
cularly  attractive  to  the  enemy  . 

In  theory,  the  enemy  can  counter  every  system  we  design  and  our 
effectiveness  will  not  be  sufficiently  high  to  warrant  a  positive  decision. 

The  real  question  is:  how  much  does  it  cost  the  enemy  in  time  and 
resources  to  effect  a  direct  counter?  If  the  price  is  very  high  he  will 
probably  seek  other  lesser  alternatives.  (See  question  6). 

50.  IS  THE  EFFECTIVENESS  MEASURED  BY  ANALYSIS  OF  WARGAMES? 

When  effectiveness  is  measured  by  analysis  of  wargames  the  review¬ 
er  should  look  to  sensitivity  analysis  of  the  results.  Asa  rule,  wargames 
are  a  questionable  means  for  measuring  effectiveness  because  of  the 
difficulty  of  testing  the  sensitivity  of  the  results.  To  do  so  means 
challenging  the  effect  of  changes  in  players,  referees,  communications, 
as  well  as  payoff  functions.  (See  question  18.) 

51.  IS  THE  EVALUATION  OF  EFFECTIVENESS  BASED  ON  STRAIGHT 

EXTRAPOLATION? 

Occasionally  a  study  may  evaluate  effectiveness  by  straight  (linear) 
extrapolation  from  the  measurement  cf  effectiveness  of  a  small  unit. 

For  example,  a  hypothetical  study  may  show  that  6  riflemen  can  destroy 
10  targets.  Am  extrapolation  that  states  100  targets  can  be  destroyed  by 
60  riflemen  is  not  justified  without  supporting  evidence.  The  variables 
in  target  and  fire  distribution  are  not  necessarily  the  same  in  both 
cases.  Further,  in  a  force  of  60  riflemen  the  percentage  who  will 
actually  fire  at  targets  may  not  be  the  same  as  for  a  force  of  6  riflemen. 

Another  error  in  straight  or  linear  extrapolation  is  disregard  of  the 
element  of  diminishing  returns  or  marginal  utility.  For  example,  200 
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missiles  do  not  signify  twice  as  much  effectiveness  as  100  missiles  if 
there  are  only  50  targets.  Furthermore,  all  targets  are  not  of  equal 
value  or  importance. 

52.  ARE  THE  OPERATIONS  OF  OTHER  SERVICES  IGNORED? 

In  measuring  the  effectiveness  of  a  system  or  organization,  consid¬ 
eration  must  be  given  to  the  operations  of  other  Services,  where  appro¬ 
priate.  Failure  to  do  so  is  the  equivalent  of  making  the  erroneous  un¬ 
stated  assumption  that  only  the  Army  will  participate  in  the  operation. 
For  example,  the  measurement  of  effectiveness  of  Army  air  defense 
operations  must  consider  the  communications,  command  and  control, 
and  IFF  aspects  of  operations  with  the  U.S.  Air  Force  and  allied  air 
forces.  Further,  the  effectiveness  cf  certain  Army  operations  is  de¬ 
pendent  upon  the  degree  of  air  superiority  achieved  by  the  Air  Force. 

The  ability  to  achieve  this  air  superiority  and  the  degree  of  dependence 
upon  it  should  be  examined.  (See  question  10. ) 

53.  IS  THE  IMPACT  ON  OTHER  ARMY  OPERATIONS  IGNORED? 

In  measuring  the  effectiveness  of  a  system  or  organization,  the 
effects  on  other  Army  operations  should  be  considered.  For  example, 
the  use  of  tactical  nuclear  weapons  in  a  certain  manner  may  accomplish 
its  function  by  stopping  enemy  ground  movement.  However,  the  judg¬ 
ment  of  the  effectiveness  of  the  system  should  also  examine  the  effect 
on  the  ground  movement  of  U.  S.  units.  In  the  same  manner  certain 
protective  clothing  may  be  effective  against  enemy  chemical  agents. 
However,  the  clothing  may  cause  such  body  heating  that  it  can  only  be 
worn  for  very  short  periods. 

54.  ARE  SOME  ASPECTS  OF  EFFECTIVENESS  INCOMMENSURABLE 

OR  UNMEASURABLE? 

The  reviewer  should  examine  carefully  the  treatment  of  incommen- 
surables  and  unmeasurable  aspects  cf  performance  in  the  total  measure¬ 
ment  of  effectiveness.  Misleading  measures  cf  effectiveness  are  now 
often  obtained  by  quantifying  such  aspects  as  morale,  or  leadership.  At 
times,  the  only  practicable  solution  may  be  a  qualitative  discussion  of 
these  factors. 

55.  DOES  THE  EFFECTIVENESS  OF  A  FUTURE  SYSTEM  TAKE  INTO 

ACCOUNT  THE  TIME  DIMENSION? 

The  effectiveness  of  proposed  future  systems  is  often  dependent 
upon  when  they  can  be  available  for  operational  use  and  the  total  opera¬ 
tional  life  span  of  the  systems.  In  examining  the  effect  cf  the  time 
dimension  upon  effectiveness,  particular  attention  should  be  given  to 
(1)  the  time  between  the  present  and  the  initial  operational  availability 
of  the  complete  system,  and  (2)  the  latter  part  of  the  system  operational 
life  span. 
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For  example,  the  effectiveness  of  Weapon  Y,  deployable  in  1972, 
is  compared  with  that  of  the  current  Weapon  X.  Weapon  Y  is  judged  to 
be  more  effective  and  requires  entirely  new  support  equipment  not 
compatible  with  that  of  Weapon  X.  This  equipment  cannot  be  operation¬ 
ally  available  until  1974.  It  is  very  possible  that  the  changeover  from 
X  to  Y  implies  a  dip  in  effectiveness  during  the  1970-74  interval.  The 
old  weapon  is  becoming  obsolete  and  the  new  one  is  not  fully  effective. 

A  quick  fix  means  may  be  needed  to  bridge  this  gap  and  must  be  charged 
to  the  cost  of  X  and  Y. 

In  another  case  involving  the  time  dimension.  Weapon  B,  deployable 
in  1972,  replaces  Weapon  A  and  is  designed  to  perform  the  same  mission 
more  effectively.  It  is  stated  to  have  an  operational  life  of  15  years. 
Effectiveness  is  calculated  on  the  basis  of  the  1972  environment.  In  the 
1972  to  1987  period  (the  operational  life  of  B)  the  international  environ¬ 
ment,  and  hence  the  missions  may  undergo  major  changes.  In  fact,  the 
mission  for  which  A  is  designed  may  already  be  on  the  decline.  Effective¬ 
ness  is  not  always  constant  but  often  must  be  related  to  time. 

It  is  necessary  to  recognize  that  missions  do  not  remain  fixed. 
Effectiveness  should  not  be  evaluated  on  the  basis  of  either  a  specific 
probability  of  the  continuity  of  the  mission  or  of  a  specified  new  mission. 
Rather,  the  system  should  bejudged  on  its  ability  to  adjust  to  such 
changes. 

Similar  comments  apply  with  respect  to  changes  in  technology. 
Breakthroughs  cannot  be  predicted  very  successfully.  Nevertheless, 
certain  trends  are  noticeable.  For  example,  anti-tank  weapons  have 
improved  more  rapidly  than  tanks  since  World  War  11.  The  sensitivity 
of  the  system  to  jumps  in  technology  is  a  vital  input  to  effectiveness 
evaluation  of  massive  long  lifetime  systems. 

56.  ARE  EXPECTED  AND  AVERAGE  VALUES  USED  INCORRECTLY 

TO  MEASURE  EFFECTIVENESS? 

It  is  an  error  to  employ  an  expected  value  or  average  as  part  of 
a  measure  of  effectiveness  if  the  objective  really  requires  a  specified 
minimum.  In  such  a  case,  the  possible  variances,  or  dispersions  about 
the  average,  consitute  an  unacceptable  risk  for  any  single  event. 

This  risk  is  unacceptable  even  though  over  many  events  the  results  will 
average  out  to  the  expected  value. 

For  example,  assume  that  at  the  same  cost,  air  defense  System  A 
destroys  from  0  to  99  of  100  approaching  enemy  aircraft  but  on  the 
average  destroys  50.  System  B,  on  the  other  hand,  destroys  from  25  to 
35  of  100  approaching  aircraft  with  an  average  destruction  of  30.  The 
risk  associated  with  the  possibility  that,  in  any  given  individual  attack 
by  100  aircraft.  System  A  may  not  destroy  any  aircraft  at  all,  whereas 
System  B  can  be  counted  on  to  destroy  at  least  25  aircraft,  makes  A  an 
unacceptable  system,  if  the  objective  is  air  defense.  If  the  objective 
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were  destruction  of  as  many  enemy  aircraft  as  possible  over  some 
period  of  time  but  without  regard  to  their  damage  to  us,  (anunlikely 
objective)  System  A  would  be  preferred. 

57.  IF  QUANTITATIVE  MEASURES  OF  EFFECTIVENESS  ARE  UN¬ 
ATTAINABLE,  IS  A  QUALITATIVE  COMPARISON  FEASIBLE? 

There  are  times  when  the  effectiveness  of  a  system  or  organization 
cannot  be  presented  adequately  in  quantitative  terms.  This  situation  is 
common  in  comparison  of  general  purpose  forces  such  as  in  studies  of 
alternative  divisions.  A  study  that  assigns  numerical  values  to  effec¬ 
tiveness  of  general  purpose  forces  should  be  examined  carefully. 

One  study  compared  alternative  divisions  in  terms  of  numerical 
scores.  Each  of  the  six  basic  factors  (firepower,  intelligence,  mobility, 
command/control/communications,  logistics,  survivability)  was  given 
a  numerical  value  and  these  values  were  summed  for  each  alternative. 

The  resulting  sums  were  compared  as  effectiveness  measures.  These 
numerical  values  are  likely  to  be  meaningless  because  the  six  basic 
factors  are  inputs  and  not  objectives.  They  combine  in  undetermined 
ways  to  make  up  the  effectiveness  of  tactics.  The  tactics,  in  turn,  com¬ 
bine  to  evolve  strategies.  For  example,  deception  tactics  strongly  in¬ 
volve  the  basic  building  blocks  of  intelligence,  command/control/commu- 
nications,  and  mobility.  However,  this  does  not  mean  that  we  can  simply 
add  up  so-called  scores  of  these  three  basic  factors  and  thereupon  com¬ 
pare  the  deception  capability  of  various  alternatives. 

A  qualitative  comparison  is  possible,  however.  Various  pertinent 
aspects  can  be  described  and  characterized  by  "yes-no"  or  "good-fair- 
poor.  "  A  tabular  comparison  can  be  useful  in  weeding  out  some  alter¬ 
natives.  It  may  be  justifiable  to  say  that  Alternative  A  is  more  effective 
than  B  (denoted^)  in  a  certain  characteristic,  even  if  it  is  not  known 
whether  A  lg  times  or  twice  as  effective  as  B.  If  it  can  be  determined 
that  ^  and  we  have  a  partial  ordering  i.  e.,  we  cannot  distin¬ 

guish  between  B  and  C  but  either  is  inferior  to’ A,  we  may  obtain  a 
grouping  as  follows: 


A 

B,  C,  D 
E 

Let  us  reconsider  the  example  of  the  deception  tactic.  Its  key  in¬ 
gredients  are  mobility,  command/control/communications,  and  intelli¬ 
gence.  .Suppose  we  know  that  Division  A  is  more  mobile  than  B,  there- 
fore,  tj-.  ff  we  should  arrive  at  the  same  ranking  for  the  other  two 
basic  factors,  then  we  conclude  that  £  is  also  true  for  the  deception  tactic. 
Cti  the  other  hand,  it  may  be  that  g-  ior  mobility  and  for  intelligence. 
Then  no  statement  can  be  made  for  the  relative  ranking  of  A  and  B  for  the 
deception  tactic . 
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In  this  manner,  tactics  of  interest  can  be  investigated  and  valid 
partial  orderings  of  alternatives  obtained.  We  may  find  dominant  alter¬ 
natives.  Suppose  we  obtain 

Command,  Control, 

Mobility  Intelligence  Communications 


A  „  B  E  .  A 

C  B,  C,  D 

D,E 


E 

A ,  B ,  C 
D 


We  have  now  learned  that  D  is  dominated  by  A  for  all  three  basic  factors, 
and  hence  for  the  deception  function.  So  D  can  be  eliminated  if  all  al¬ 
ternatives  have  equal  cost.  It  should  not  be  assumed  that  rankings  of 
alternatives  with  respect  to  the  tactical  level  can  only  be  derived  by 
buildup  and  integration  from  the  basic  level.  There  may  be  direct  quali¬ 
tative  comparisons  with  respect  to,  say,  deception  effectiveness  as  a 
result  of  wargames  or  field  exercises.  A  combination  of  both  buildup 
and  direct  approaches  would  probably  prove  most  fruitful. 

The  reviewer  should  recognize  that  while  cost-effectiveness  analysis 
is  performed  preferably  by  quantitative  analysis,  there  are  limits  to 
suboptimizing  or  idealizing  the  problem  to  make  it  amenable  to  quantita¬ 
tive  analysis.  When  carried  too  far,  the  quantitative  results  are  often 
only  of  academic  interest  and  offer  little  or  no  help  to  the  decision  maker. 

58.  IS  THE  EFFECTIVENESS  SENSITIVE  TO  CHANGES  IN  ASSUMPTIONS? 


The  effectiveness  derived  in  most  studies  is  usually  dependent  to  a 
degree  on  the  assumptions.  The  reviewer  should  isolate  the  degree  of 
dependence  and  determine  if  it  is  acceptable.  Generally, 
a  good  study  will  isolate  this  dependence,  where  it  exists,  and  lay  out 
the  degree  of  dependence  by  various  kinds  cf  sensitivity  or  contingency 
analysis.  The  assumptions  that  most  commonly  influence  effectiveness 
and  are  often  not  subjected  to  contingency  analysis  concern  the  locale, 
the  time  and  level  of  warfare,  and  enemy  forces  and  tactics. 

A  slight  change  in  any  cf  these  assumptions  may  produce  significant 
changes  in  the  effectiveness  measured.  For  example,  additions  cf  a  new 
ECM  band  width  to  the  enemy’s  capability  may  drastically  degrade  an 
otherwise  outstanding  U.  S.  system.  (See  questions  5  and  6.  ) 
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CRITERIA 


59.  ARE  THE  CRITERIA  IDENTIFIED? 

The  criteria,  or  tests  cf  preferredness,  are  the  basis  for  the  con¬ 
clusions  and  recommendations.  The  criteria  should  be  stated  specifi¬ 
cally  and  clearly.  IF  this  is  not  the  case,  the  reviewer  should  attempt 
to  identify  the  criteria  from  the  material  contained  in  the  study.  When 
this  does  not  prove  possible,  consideration  should  be  given  to  having  the 
study  returned  for  futher  clarification.  This  is  particularly  important 
if  the  study  is  also  to  be  reviewed  by  agencies  outside  the  Army. 

60.  ARE  THE  CRITERIA  CONSISTENT  WITH  HIGHER  ECHELON 

OBJECTIVES  ? 

No  matter  what  the  concern  cf  a  study,  the  subject  falls  into  a 
larger  framework.  For  example,  problems  of  air  defense  of  the  CONUS 
are  aspects  of  the  larger  problem  of  restricting  possible  damage  to  the 
CONUS  to  certain  levels.  The  design  of  artillery  systems  is  part  of  the 
larger  problem  of  design  of  land  battle  forces.  Therefore,  the  reviewer 
must  determine  if  the  criteria  used  in  a  study  are  consistent  with  higher 
level  objectives.  This  requires  good  military  judgment  and  the  necessity 
to  examine  the  larger  context  of  the  problem.  If  the  study  criteria  are 
not  consistent  with  the  objectives  at  the  higher  level  then  the  wrong 
problem  may  be  solved.  Overall  Army  objectives  are  contained  in  docu¬ 
ments  such  as  the  Basic  Army  Strategic  Estimate  (BASE),  Army  Force 
Development  Plan  (AFDP),  Army  Strategic  Plan  (ASP),  and  the  Combat 
Developments  Objective  Guide  (CDOG). 

An  example  of  incorrectly  chosen  criteria  is  illustrated  in  the  use 
of  mobility  as  the  sole  criterion  in  the  selection  among  different  organi¬ 
zations.  A  study  could  conceivably  demonstrate  that  organization  A  can 
be  made  more  mobile  than  organizations  B  and  C  with  a  lesser  expendi¬ 
ture  of  resources.  Yet  A  may  not  be  the  preferred  organization  because 
the  mobility  was  achieved  by  degrading  other  factors  that  contribute  to 
the  higher  objective  of  efficient  control  cf  conflict  situations  (e.g.  ,  fire¬ 
power  ,  sustainability  ,  etc.  ). 

61.  ARE  THE  CRITERIA  TOO  GENERAL? 

Generalized  criteria  are  suspect.  For  example,  a  study  may  state 
that  the  criterion  is  "the  system  with  maximum  military  worth"'  or  the 
"best  system".  Such  generalizations  are  meaningless  and  cannot  be 
related  to  the  analysis  as  can  a  good  criterion  such  as  "the  minimum 
cost  of  maintaining  a  [  specified]  level  cf  transport  capability  over  a 
[  specified]  time  span.  " 
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62.  ARE  THE  CRITERIA  OVERDETERMINED? 

Overdetermined  criteria  lead  to  erroneous  conclusions.  A  criterion 
that  states  "to  maximize  the  damage  to  the  enemy  while  at  the  same  time 
minimizing  the  cost  to  the  U.  S."  or  "causing  the  maximum  amount  of 
casualties  with  the  least  expenditure  of  ammunition"  suggests  that  some¬ 
thing  can  be  obtained  for  nothing.  It  is  impossible  to  maximize  gain  and 
simultaneously  minimize  cost.  It  is  not  possible  to  increase  effectiveness 
without  some  increase  in  resources  (cost).  The  minimum  cost  is  to  do 
nothing — and  achieve  no  effectiveness.  Occasionally  it  may  turn  out  that 
system  A  is  both  more  effective  than  system  B  and  costs  less.  However, 
system  A  will  not  be  both  more  effective  and  cost  less  when  compared 
with  additional  alternatives.  The  danger  of  using  an  overdetermined 
criterion,  such  as  the  one  described,  is  that  it  leads  to  invalid  compro¬ 
mise  criteria  by  using  some  erroneous  constraint  on  effectiveness  or 
cost  in  an  effort  to  make  an  impossible  test  seem  feasible. 

63.  ARE  GOOD  CRITERIA  APPLIED  TO  THE  WRONG  PROBLEMS? 

At  times  a  valid  criterion  for  one  element  cf  the  problem  is  incor¬ 
rectly  applied  to  the  total  problem.  For  example,  a  hypothetical  study 
involving  proposed  surveillance  aircraft  shows  that  aircraft  A  offers 
greater  mission  flexibility  than  aircraft  B  at  the  same  cost  and  is  there¬ 
fore  preferred.  In  this  case,  the  choice  of  aircraft  is  not  the  real 
problem.  The  subsystems  carried  by  the  aircraft  are  really  more 
crucial.  The  all-weather  sensor  effectiveness  and  avionics  cost  may 
even  determine  whether  there  should  even  be  an  aircraft  A  or  B. 

64.  IS  THE  ABSOLUTE  SIZE  OF  GAIN  OR  COST  INGNORED? 

F  the  absolute  size  of  the  cost  of  a  system  alternative,  or  the 
effectiveness  to  be  achieved  by  it,  is  not  given  or  is  incorrect,  the 
analysis  often  leads  to  wrong  conclusions  and  recommendations.  For 
example,  cost-effectiveness  curves  for  two  hypothetical  system  alter¬ 
natives  are  given  below: 


ENEMY  TARGETS  DESTROYED 
(EFFECTIVENESS) 


Figure  2.  Cost-Effectiveness  Curves 
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In  this  situation,  at  low  levels  of  effectiveness,  alternative  A  is 
preferred  (up  to  about  70  enemy  targets  destroyed);  at  larger  levels 
of  effectiveness  alternative  B  is  preferred  (from  about  70  to  110).  If 
the  capability  to  destory  more  than  110  enemy  targets  is  to  be  achieved, 
then  neither  alternatives  A  or  B  is  preferred  or  even  acceptable.  The 
crucial  question  is  how  many  enemy  targets  are  required  to  be  destroyed. 
If  the  number  of  enemy  targets  to  be  destroyed  or  cost  limits  are  not 
indicated,  there  is  no  real  basis  to  recommend  either  alternative  A  or 
B,  or  some  other  alternative. 

Either  the  study  should  be  based  on  an  absolute  size  of  gain  or  cost 
required  or  the  study  should  present  a  cost-effectiveness  curve  (or  points) 
from  which  decisions  can  be  made.  If  the  study  presents  a  cost-effective¬ 
ness  curve  as  shown  above,  the  envelope  (indicated  by  line  of  X’ s  in  the 
graph)  along  the  bottom  says,  "This  is  a  curve  which  gives  the  most  for 
the  resources  expended,  and  other  things  have  to  be  taken  into  consider¬ 
ation  at  higher  levels  to  determine  what  the  absolute  gain  (number  of 
targets  destroyed)  should  be  or  the  maximum  resources  (cost)  that  can  be 
made  available.  " 

At  times,  studies  ignore  absolute  size  of  gain  or  cost  and  use  effec¬ 
tiveness-to-cost  ratios.  The  flaw  in  the  use  of  such  ratios  is  the  absence 
of  any  specified  level  of  effectiveness  required  or  resources  available 
as  discussed  above.  If  the  level  of  activity  is  fixed,  a  ratio  may  be  use¬ 
ful  in  ranking  among  alternative  systems.  However,  the  effectiveness- 
to-cost  ratio  criterion  is  often  applied  when  the  level  of  activity  is  not 
fixed.  For  example,  in  the  graph  above  alternative  A  destroys  10  enemy 
targets  for  $1  million,  and  alternative  B  destroys  100  enemy  targets  for 
$25  million.  If  only  this  information  were  converted  to  effectiveness-to- 
cost  ratios,  alternative  A  would  have  a  ratio  of  10:  land  alternative  B, 

4:1.  Which  is  the  preferred?  If  one  did  not  look  at  the  absolute  level 
of  effectiveness  required  to  achieve  the  military  task  but  only  at  the 
effectiveness-to-cost  ratio,  then  alternative  A  would  be  preferred.  The 
selection  of  alternative  A  on  this  basis  may  be  correct,  but  only  by 
coincidence  and  is  obviously  wrong  when  the  system  must  be  capable  of 
destroying  more  than  70  targets. 

Until  the  absolute  level  (magnitude)  of  effectiveness  or  the  absolute 
level  (magnitude)  of  the  cost  is  specified  the  preferred  alternative  cannot 
be  determined.  The  effectiveness-to-cost  ratio  can  be  misleading  and, 
at  times,  a  dangerous  criterion. 
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CONCLUSIONS  AND  RECOMMENDATIONS 


65.  ARE  THE  CONCLUSIONS  AND  RECOMMENDATIONS  LOGICALLY 

DERIVED  FROM  THE  MATERIAL  CONTAINED  IN  THE  STUDY? 

The  conclusions  and  recommendations  should  be  derived  logically 
from  the  material  contained  in  the  study.  Some  studies,  unfortunately, 
draw  conclusions  based  on  previous  studies  and  materials  that  are  not 
fully  documented  within  the  study  (mention  in  a  bibliography  is  hardly  suf¬ 
ficient)  .  ff  input  from  another  study  is  essential,  it  should  be  documented 
and  explained  in  detail.  This  requires,  at  least,  a  statement  as  to  validity, 
scope  of  application  and  uncertainty  which  is  associated  with  the  particular 
input . 

The  determination  of  whether  the  conclusions  and  recommendations 
follow  logically  from  the  material  in  the  study  is  a  matter  cf  judgment  by 
the  reviewer  .  In  making  this  judgment,  the  reviewer  should  consider 
whether  other  prudent  etudy  agencies  would  probably  arrive  at  substan¬ 
tially  the  same  conclusions  and  recommendations  given  only  the  material 
contained  in  the  study. 


66.  HAVE  ALL  THE  SIGNIFICANT  RAMIFICATIONS  BEEN  CONSIDERED 
IN  ARRIVING  AT  THE  CONCLUSIONS  AND  RECOMMENDATIONS 
CONSIDERED? 

Sometimes  a  study  fails  to  consider  all  the  pertinent  ramifications  in 
arriving  at  the  conclusions  and  recommendations  .  These  unconsidered 
ramifications  may  either  influence  the  validity  of  the  conclusions  and 
recommendations  of  the  study  or  the  decisions  to  be  made  as  a  result  of 
the  study.  These  ramifications  are  often  referred  to  as  "  spillovers.  " 

For  example,  if  a  hypothetical  study  recommended  adoption  of  an  engine 
requiring  a  new  type  of  fuel,  the  Army  supply  system  to  include  supply, 
storage  and  transportation  operations  would  be  affected.  Spillover  effects 
are  not  always  negative.  For  example,  the  adoption  cf  dehydrated  rations 
to  achieve  greater  shelf-life  may  also  reduce  construction  and  transporta¬ 
tion  costs  because  of  the  smaller  unit  volume  cf  dehydrated  food. 

Other  ramifications  that  are  sometimes  neglected  are  factors  that 
should  be  considered  jointly  with  the  problem  under  study.  At  times,  con¬ 
sideration  of  such  joint  decisions  could  affect  the  conclusions  and 
recommendations  of  the  study.  For  example,,  a  study  may  recommend 
adoption  of  a  new  weapon  system  to  fulfill  a  certain  function.  However,  the 
study  may  neglect  to  examine  the  maintenance  support  and  the  maintenance 
units  that  would  have  to  be  in  existence  concurrently  with  the  proposed 
weapon  system.  The  resources  required  to  organize  and  maintain  the 
maintenance  system  will  influence  decisions  on  the  proposed  weapon  system. 
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If  significant  ramifications  are  uncovered  that  are  not  adequately  con¬ 
sidered,  the  reviewer  should,  if  possible,  determine  the  effects  cf  these 
ramifications  on  the  conclusions  and  recommendations  .  (See  question  2.  ) 


67.  ARE  THE  CONCLUSIONS  AND  RECOMMENDATIONS  REALLY 

FEASIBLE  IN  THE  LIGHT  OF  POLITICAL,  CULTURAL,  POLICY  OR 
OTHER  CONSIDERATIONS? 

In  reviewing  the  conclusions  and  recommendations  of  a  study,  it  is 
necessary  to  be  cognizant  of  the  real  world  in  which  the  Army  must  operate. 
At  times  some  recommendations  of  a  study  may  appear  to  be  eminently 
feasible  from  a  strictly  economic  or  military  view,  but  really  are  not  SO 
in  the  light  of  other  considerations  that  influence  military  operations.  For 
example,  a  particular  toxic  chemical  munitions  system  may  be  demon¬ 
strated  to  be  superior,  considering  cost  and  effectiveness,  to  a  high 
explosive  munitions  system  for  accomplishing  a  certain  function.  However, 
because  of  national  policies  on  employment  cf  toxic  chemicals,  the  adoption 
cf  the  high  explosive  munitions  system  may  be  the  only  feasible  solution. 

The  reviewer  should  also  consider  the  impact  of  policies  that  may  not 
have  been  known  to  the  agency  that  prepared  the  study  or  were  promul¬ 
gated  too  late  to  influence  the  study. 


68.  ARE  THE  CONCLUSIONS  AND  RECOMMENDATIONS  RELATED  TO 

THE  LIMITATIONS  OF  THE  STUDY? 

In  evaluating  conclusions  and  recommendations,  the  reviewer  should 
bear  in  mind  the  limitations  cf  the  study  .  Studies,  as  a  rule,  have 
varying  degrees  cf  limitations.  The  more  common  types  of  limitations 
include  inadequate  data  base,  criticality  cf  assumptions,  criticality  of 
uncertainties  and  validity  of  cost  and  effectiveness  models.  While  the 
limitations  may  be  treated  within  the  study,  the  dependence  of  the  con¬ 
clusions  and  recommendations  on  the  limitations  is  sometimes  neglected. 
For  example,  the  study  conclusions  and  recommendations  may  depend 
upon  the  validity  of  particular  assumptions  but  this  relation  may  not  be 
pointed  out. 

It  may  be  advisable  for  the  reviewer  to  refresh  his  memory  on  the 
study  limitations,  particularly  when  the  study  is  voluminous,  before 
evaluating  the  conclusions  and  recommendations. 

69.  DO  THE  CONCLUSIONS  AND  RECOMMENDATIONS  INDICATE  BIAS? 

Studies  sometimes  unwittingly  reflect  bias  because  cf  parochial  or 
institutional  interests.  To  assist  in  detecting  bias,  the  reviewer  should 
consider  the  relation  of  the  agency  that  prepared  the  study  and  the  effects 
of  the  implementation  c£  the  study  recommendations  .  If  such  implementa¬ 
tion  does  not  appear  to  further  what  are  generally  considered  to  be  the 
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particular  interests  of  the  preparing  agency,  then  one  occasional  form  of 
bias  is  probably  not  present.  Another  test  for  bias  is  to  judge  whether 
substantially  the  same  conclusions  and  recommendations  would  be  reached, 
based  on  the  material  in  the  study,  by  another  study  agency.  Bias  is 
often  displayed  by  arbitrarily  excluding  certain  reasonable  alternatives, 
maximizing  selected  enemy  capabilities,  treating  significant  uncertainties 
as  assumptions,  and  in  selection  of  effectiveness  criteria- 

A  relatively  minor  form  of  bias  is  sometimes  found  in  the  use  of 
prejudicial  adjectives.  Unnecessarily  referring  to  all  Air  Force  fixed 
wing  aircraft  as  "  long  take  off  and  landing"  aircraft  is  an  example.  This 
type  of  bias  may  be  prejudicial  to  the  interests  of  the  Army  when  the  study 
is  reviewed  by  non-Army  agencies. 


70.  ARE  THE  CONCLUSIONS  AND  RECOMMENDATIONS  BASED  ON 

EXTERNAL  CONSIDERATIONS? 

At  times,  recommended  selections  among  alternatives  must  be  made 
in  the  face  cf  great  uncertainty.  A  study  may  find  that  several  alterna¬ 
tives  exhibit  similar  cost-effectiveness,  but  the  results  are  very  sensitive 
to  the  values  assigned  to  the  inputs.  In  this  situation  some  studies  arrive 
at  conclusions  and  recommendations  based  on  considerations  other  than 
those  studies.  In  other  words,  the  study  agency  is  stating  that  after 
having  made  the  analysis,  the  application  of  the  criteria  does  not  lead  to 
preference,  but  indifference,  among  the  alternatives  and  therefore  the 
issue  was  decided  on  the  basis  of  other  unstudied  criteria.  In  situations 
of  this  kind,  when  recommendation  of  an  alternative  is  necessary,  sensi¬ 
tivity  to  new  criteria  must  be  fully  studied. 


7 1 .  ARE  THE  CONCLUSIONS  AND  RECOMMENDATIONS  BASED  ON 

INSIGNIFICANT  DIFFERENCES? 

At  times  a  study  will  present  one  alternative  as  having  the  highest 
value  of  effectiveness  of  the  measures  applied  to  all  alternatives.  The 
difference  in  effectiveness  among  the  "  optimum"  alternative  and  the 
other  alternatives  should  be  examined.  F  the  differences  are  relatively 
slight  and  probably  no  greater  than  the  uncertainties  in  the  data,  then 
other  grounds  should  also  be  demonstrated  for  selecting  among  the  alterna¬ 
tives  that  are  close  in  effectiveness. 


72.  IF  PRIORITIES  ARE  LISTED,  ARE  THEY  STATED  MEANINGFULLY? 

Conclusions  and  recommendations  often  list  items  cf  equipment  in 
order  of  priority  of  recommended  procurement  or  adoption.  The  use  cf 
this  technique  without  explanation,  particularly  for  materiel  ,  is  often 
poor  because  it  provides  no  basis  for  a  decision.  For  example,  a  study 
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may  conclude  that  in  order  to  accomplish  certain  functions,  infantry  units 
should  be  equipped  with  specified  items  of  equipment  that  are  listed  in 
order  of  priority.  Assume  that  the  items  found  necessary  by  the  study  for 
infantry  units  to  accomplish  the  required  functions  are,  in  order  of 
priority : 

(1)  Seven  League  Boots 

(2)  Disintegrator  Ray  Pistol 

(3)  Universal  Viewing  Device 

(4)  Camouflage  Suit  (makes  the  wearer  invisible) 

Although  the  study  concluded  that  all  of  these  items  are  required,  the 
listing  of  priorities  without  any  quantitative  considerations  could  have  any 
of  these  meanings: 

a.  Buy  all  of  the  Seven  League  Boots  required.  Then,  as  resources 
are  available,  buy  all  of  the  Disintegrator  Ray  Pistols  required.  Continue 
down  the  list  of  priorities  in  this  manner  until  the  available  resources  are 
exhausted.  This  meaning  also  infers  that  even  though  all  4  items  are  re¬ 
quired,  the  Army  can  do  without  the  lower  priority  items  if  sufficient 
resources  are  not  available  to  procure  them  all.  For  example,  with 
limited  resources  it  is  better  to  have  all  Seven  League  Boots  and  none  cf 
the  other  items  rather  than  some  of  each  item. 

b.  Buy  all  4  items  at  once  but  spend  more  money  on  Seven  League 
Boots  than  on  Disintegrator  Ray  Pistols  and  even  less  amounts  on  Universal 
Viewing  Devices  and  Camouflage  Suits. 

When  faced  with  this  kind  of  situation,  consideration  should  be  given 
to  returning  the  study -to  the  preparing  agency  for  further  recommendations 
on  how  much  should  be  allocated  to  each  item  for  various  budget  levels, 
either  given  or  assumed. 


73.  ARE  THE  CONCLUSIONS  AND  RECOMMENDATIONS  INTUITIVELY 
SATISFYING? 

When  the  conclusions  and  recommendations  of  the  study  are  not 
intuitively  satisfying,  the  reviewer  should  attempt  to  isolate  the  cause, If 
the  study  fails  to  demonstrate  by  data,  models  and  other  means  that  the 
reviewer's  intuition  was  wrong,  then  further  examination  is  required  to 
determine  if  some  subtle  considerations  have  not  been  considered  because 
of  oversimplification  or  other  reasons  which  the  reviewer  intuitively  knows 
are  pertinent. 
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AN  EXAMPLE  OF  A  RECENT  COMMUNICATIONS 
COST-EFFECTIVENESS  ANALYSIS  * 

THE  PROBLEM 


Demonstrate  the  feasibility  of  the  commimication-system  cost-effectiveness 
method  developed  and  reported  previously  in  NEL  report  1323. 1 


RESULT 


1.  The  communication-system  cost-effectiveness  method  has  been  comput¬ 
erized  and  applied  to  the  analysis  of  a  shore-to-ship  hf  link. 

a.  Sensitivity-analysis  curv  es  are  developed  for  the  variable  parameters 
used  hi  the  analysis. 

b.  Cost-estimating  equations  have  been  developed  for  the  basic  equip¬ 
ments  comprising  an  hf  communication  link. 

c.  Trade-off  curves  illustrating  system  cost  versus  system  parameters, 
such  as  propagation  loss,  effective  antenna  noise  figure,  and  error  rates,  are 
provided. 


RECOMMENDATIONS 


1.  Establish  a  central  Navy-wide  data  storage  and  retrieval  system  to  stand¬ 
ardize  and  maintain  historical  and  current  data  to  support  future  resource-effec¬ 
tiveness  analysis. 

2.  Extend  the  method  to  include  the  communication  system  in  the  operational 
environment. 

3.  Develop  a  local  data  storage  and  retrieval  program  to  support  further 
resource-effectiveness  analysis . 


ADMINISTRATIVE  INFORMATION 


Wak  was  perforated  under  XF  006  01  07,  Task  7592  (NEL  B20871),  by 
members  of  the  Communication  Techniques  Division.  The  report  covers  work 
from  June  1965  to  September  1966  and  was  approved  for  publication  4  January  1967. 


See  reference*  in  appendix 


*Cost  Effectiveness  AR&kys^s  Applied  to  H  F  Communications  Link  by  W.  R.  Dishong, 
Jr. ,  and  R  T!  Marloth.,  4  January  1967;  NEL  Report  1422 ;  Reproduced  by  per¬ 
mission  of  U,  S.  Navy  Electronics  Laboratory,  San  Diego,  Calif. 
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INTRODUCTION 


Historical  Background 

This  report  summarizes  the  continuation  cf  the  work  in  analysis  of  com¬ 
munication-system  effectiveness  reported  in  NEL  report  1323. 1 

Mary  reports  such  as  the  foregoing  are  available  to  guide  analysts  hi 
evaluating  and  analyzing  various  system  configurations.  Some  of  the  reported 
methods  call  for  many  judgment  decisions  on  the  part  cf  the  analyst,  and  in 
some  cases  require  cost  data  that  are  impossible  to  obtain.  Also,  some  methods 
permit  grouping  all  the  system  characteristics  into  a  single  “figure  cf  merit,” 
or  measure  of  effectiveness.  The  latter  method  is  often  misleading  and  meaning¬ 
less  to  the  system  analyst  and  decision  maker.  Other  methods  allow  system 
effectiveness  to  be  specified  in  terms  cf  multiple  measures  cf  effectiveness. 
Several  reasons  for  using  multiple  measures  cf  effectiveness  are  contained  in 
reference  1,  as  well  as  in  appendix  A. 

Single  methods  are  not  used  in  this  report,  as  they  are  considered  un¬ 
realistic  in  the  evaluation  of  complex  systems  that  have  multiple  objectives. 

Another  method  for  analysis  of  cost  effectiveness  is  one  that  realistically 
represents  the  pertinent  system  characteristics  and  costs,  is  capable  of  being 
implemented,  and  can  be  used  in  a  meaningful  maimer  by  the  system  analyst  and 
decision  maker.  (Unfortunately,  the  more  realistic  and  representative  a  model 
is,  the  more  difficult  it  is  to  implement,  and  often  compromises  have  to  be  made 
because  of  the  time  required  for  analysis.)  It  is  this  method  that  is  being 
developed. 


Statement  of  Problem 


The  ultimate  objective  of  the  work  in  communication  effectiveness  is 
twofold '  '  to  develop  a  communication-system  resource-effectiveness  method, 
and  to  perfonn  resource-effectiveness  analysis  of  communication  systems. 

The  method  is  intended  for  use  as  a  management  tool  and  as  a  design 
tool.  Asa  management  tool,  the  method  can  be  used  in  the  preparation  of 
Proposed  Technical  Approaches  (PTA),  Technical  Development  Plans  (TDP), 
and  Detailed  Action  Plans  (DAP)  to: 

1.  Determine  resource  effectiveness  of  a  set  cf  technical  approaches, 

2.  Establish  performance  estimates,  and 

3.  Conduct  resource-effec  tiveness  trade-off  studies. 

As  a  design  tool,  the  method  can  be  used  to: 

1.  Specify  system  characteristics, 

2.  Specify'  system  effectiveness,  and 

3.  Specify  system  resource  requirements. 

This  report  summarizes  the  work  accomplished  to  date  in  implementing  the 
cost-effectiveness  method  developed  and  documented  in  NEL  report  1323.  The 
method  as  developed  to  date  was  applied  to  a  real  communication  situation  to 
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detennine  its  validity.  An  hf  Fleet  broadcast  link  was  selected  because  of  the 
availability  of  data  pertaining  to  hf  equipments  and  other  studies  concerning  this 
type  of  link.  The  performance  model  was  used  to  evaluate  and  analyze  an  hf 
Fleet  broadcast  link  with  respect  to  performance  and  cost.  The  analysis  illus¬ 
trates  the  contribution  of  various  system  parameters  to  overall  system  perfor¬ 
mance.  A  particular  geographical  hf  link  was  selected  for  the  analysis  to  provide 
propagation  losses,  noise  levels,  operating  frequency,  and  antenna  gain  as 
functions  of  radiation  path. 

Primary  effort  to  date  has  been  directed  to  the  system  performance  model 
and  its  cost  analysis.  During  the  course  cf  this  work,  it  became  obvious  that 
the  parameter  “cost”  as  used  in  cost  effectiveness  should  be  more  general  and 
should  include,  in  addition  to  dollar  cost,  such  items  as  material,  time,  and 
personnel.  The  more  general  descriptor  “resource”  will  be  used  in  place  cf 
“cost”  in  order  to  consider  these  items  in  proper  perspective.  In  most  cases, 
dollars  will  still  be  involved  in  the  analysis.  Flowever,  the  limitation  cf  asso¬ 
ciating  everything  with  dollars  is  removed. 


METHODS 


Problems  in  Cost-Effectiveness  Analysis 

The  following  comments  on  techniques  for  the  analysis  of  cost  effective¬ 
ness  draw  from  the  content  of  related  references  2  through  5  as  well  as  from 
experience  gained  during  the  course  cf  this  investigation.  The  uninitiated  are 
likely  to  think  of  cost  effectiveness  as  a  method  that  maximizes  effectiveness 
while  minimizing  cost.  This  conception  is  overly  optimistic.  Whal  cost- 
effectiveness  methods  can  do  is  to  minimize  cost  of  a  system  while  holding 
effectiveness  to  some  minimum  acceptable  level  or  to  maximize  effectiveness 
while  holding  cost  to  some  maximum  acceptable  level.  The  possibility  arises 
that  the  analysis  will  be  misleading  if  the  wrong  fixed  levels  are  chosen  for 
cost  or  effectiveness.  A  way'  around  this  difficulty'  is  to  compare  the  results  of 
analysis  at  several  levels  of  cost  and/or  effectiveness. 

The  optimization  should  be  executed  for  the  system  as  a  whole.  If  the 
subsystems  are  optimized  individually  (“suboptimization”),  the  result  will 
most  likely  not  be  true  optimization.  For  instance,  the  best  receiver  for  some 
given  cost  might  have  high  sensitivity  and  low  stability,  but  the  best  system 
might  call  fora  receiver  with  moderately  good  values  cf  each. 


PROBABILITY  LIMITATIONS 

The  single-link  problem  can  be  expressed  in  terms  of  physical  parameters 
rather  than  probabilistic  terms.  This  method  avoids  the  problems  associated 
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with  the  use  of  unverified  probabilities.  Solutions  to  the  problems  associated 
with  unverified  probabilities  will  be  sought  during  subsequent  investigations. 

An  example  cf  an  unverified  probability  is  in  the  answer  to  this  question: 

What  is  the  probability  that  two  ships  will  be  transmitting  on  the  same  frequency 
simultaneously?  Sometimes  questions  like  this  one  cannot  be  avoided.  If 
they  cannot,  the  sensitivity  of  the  analysis  to  the  assumed  probability  should 
be  tested  fcy  analyzing  the  system  for  various  values  cf  the  probability  under 
consideration.  If  the  analysis  turns  out  to  be  sensitive  to  the  probability,  the 
results  should  be  considered  skeptically.  One  tempting  way  out  cf  die 
unverified-probability  impasse  is  to  accept  the  word  cf  some  outside  authority 
on  die  mader;  but  thi  s  is  merely  a  way  of  avoiding  the  chore  cf  reconciling  the 
analysis  to  this  problem,  since  it  is  obvious  that  if  the  probability  is  unverifiable, 
the  expert  has  no  more  assurance  of  its  correctness  than  the  analyst  does. 

Again,  in  treating  matters  cf  probability7,  the  analyst  must  realize  that  the 
most  likely  event  does  not  always  happen.  Low-’probability  eventualities  should 
be  inspected,  too,  for  disastrous  outcomes.  When  the  real  situation  becomes  so 
complex  that  it  cannot  be  analyzed  directly,  a  simplified  model  is  used,  with 
the  danger  that  the  analyst  will  become  more  interested  in  the  model  dian  in  die 
real  situation.  Game  theory7  is  deplored  ty  many  because  it  is  so  often  invoked 
for  simple  models,  but  is  so  difficult  to  apply  to  complex  real-life  situations. 


DATA  LIMITATIONS 

A  problem  which  this  study  encountered  was  the  limited  cost  data  avail¬ 
able.  This  is  a  warning  that  the  cost  equations  obtained  by7  the  regression 
analysis  will  not  be  as  accurate  as  desirable.  Under  these  circumstances  it  is 
especially  dangerous  to  extrapolate  the  results  to  ranges  of  the  variables  for 
which  there  are  no  data. 

In  the  same  vein,  it  is  a  mistake  to  ignore  a  variable  which  cannot  be 
quantized;  for  example,  ease  of  operation  of  equipment.  It  is  always  easier  for 
the  analyst  to  insert  the  effect  of  a  quantifiable  variable  into  a  performance 
equation  than  to  philosophize  upon  the  effect  of  an  unquantifiable  one.  but  the 
tendency  to  ignore  unquantifiable  factors  should  be  controlled. 

When  the  present  study  is  expanded  to  include  more  complex  problems  of 
evaluating  large  communication  systems  to  be  built  in  the  future,  the  costing 
methods  will  have  to  be  modified  and  extended.  Some  cf  the'eonsiderations  which 
will  become  important  are  the  period  over  which  the  cost  of  new  equipment  is 
amortized,  the  difference  between  sunk  costs  (money  already  spent)  and  future 
costs,  the  significance  of  cost  differences,  and  ways  to  attach  dollar  costs  to 
training  of  personnel  and  other  expenses  not  directly  for  equipment. 


LIMITATIONS  OF  SINGLE-PARAMETER  ANALYSIS 

Most  systems  are  not  so  simple  that  their  effectiveness  can  be  expressed 
as  a  single  parameter.  If  they  were,  cost-effectiveness  analyses  would  be  much 
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more  straightforward  than  they  are.  For  example,  if  the  effectiveness  of  a 
communication  system  were  equated  with  information  rate,  all  considerations  of 
reliability  and  maintainability  would  be  sacrificed. 

Combining  the  effectiveness  measures  of  a  system  into  a  single  “quality 
factor”  is  also  specious.  For  a  time,  this  was  a  popular  practice,  but  now  it  is 
discouraged  by  many  practiced  analysts.  Appendix  A  illustrates  one  problem 
associated  with  defining  effectiveness  as  a  single  measure.  A  particularly 
strong  statement  on  the  subject  was  made  ty  E.  S.  Quade:3  “One  thing  we 
cannot  do  is  construct  Iran  all  the  individual  objectives  some  group  objective 
by  appropriately  weighing  all  separate  ones;  this  is  a  practical  absurdity  and  it 
has  been  theoretically  demonstrated  that  there  is  no  unique  and  satisfying  way 
to  do  it.”  For  example,  consider  two  systems  with  performance  factors,  A,  B. 
and  C  of  equal  importance  (weight)  evaluated  on  a  scale  fiom  1  to  5: 

Factor  A  B  C  I  Sum  Product 


System  I  4  1  5  1  1  0  20  Qt 


System  II 


3 


4 


24  Qn 


If  the  quality  factor  Q  =  A  t  B  t  C,  system  I  has  the  higher  Q;ifO  =  AxBxC, 
system  II  does.  And  there  is  no  way  of  deciding  which  fonnula  for  Q  (if  either) 
is  legitimate.  Furthermore,  deciding  upon  the  relative  importance  of  the  perfor¬ 
mance  factors  is  an  arbitrary'  process  with  serious  consequences.  If  the  pro¬ 
duct  fonn  of  Q  is  used,  weighting  does  not  affect  the  ratio  Qj/Qn-  However, 
when  the  sum  fonn  of  Q  is  used,  giving  factor  B  a  relative  weight  of  2  gives 
system  II  the  advantage  of  13  to  11.  Likewise,  system  I  has  a  15  to  11  advan¬ 
tage  if  factor  C  is  given  a  relative  weight  of  2.  Finally,  the  evaluations  are 
sensitive  to  small  changes  in  the  values  of  the  performance  factors.  If  these 
are  not  specifically  quantifiable,  the  danger  of  upsetting  the  results  is  great; 
for  example,  if  factor  B  of  system  I  is  changed  fiom  1  to  2,  the  product  Q  is 
doubled  and  far  outstrips  the  comparable  Q  of  system  11. 

After  all  the  arbitrary'  manipulations  are  made,  the  probability'  is  sur¬ 
prisingly  high  that  the  (product) Q  factors  of  the  two  systems  will  be  equal,  as 
shown  in  Appendix  A. 


System  Effectiveness  Model 

If  the  description  of  a  hy  pothetical  communication  situation  is  given, 
system  requirements  can  be  specified.  The  requirements  determine  the  objec¬ 
tives  cf  the  system.  Communication-system  objectives  may'  be  many  and  varied. 
Some  typical  communicat ion-system  objectives  are: 

1.  Information  reliability' 

2.  Information  rate 

3.  System  reliability' 

4.  System  availability' 

5.  Anti-DF 

6.  Anti -jam 
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This  report  considers  only  the  first  two  objectives.  Subsequent  work  will  inte¬ 
grate  other  objectives  into  the  overall  model.  System  objectives  may  hi  many 
cases  be  considered  as  measures  of  system  effectiveness.  Cost  as  a  resource 
element  is  not  considered  a  measure  of  effectiveness.  Cost  is  a  criterion  for 
choosing  between  alternative  systems  at  some  specified  effectiveness  level. 

The  RESULTS  section  illustrates  how  cost  can  also  be  used  to  select  between 
alternative  systems  that  exceed  the  minimum  requirements  to  different  degrees. 

Figure  1  illustrates  the  procedure  followed  in  evaluating  system  resource 
requirements  and  effectiveness. 

The  resource-effectiveness  analysis  begins  with  a  given  set  of  system 
requirements  as  shown  in  figure  1.  From  the  requirements,  mission  objectives 
are  specified.  The  mission  objectives  indicate  the  type  of  optimization  to  be 
sought;  that  i  s ,  the  most  effective  system  for  a  given  level  of  funding  or  the 
least  expensive  system  for  a  specified  level  of  effectiveness.  The  type  of 
objective  is  also  reflected  in  the  constraints  imposed  on  the  optimization  pro¬ 
cedure.  The  system  model  is  an  analytical  model  of  the  system  intended  to  ful¬ 
fill  the  stated  objectives.  The  system  model  interrelates  system  characteristics 


REQUIREMENTS  & 
EFFECTIVENESS 
OF  DESIGN 
ALTERNATIVES 


t 


Figure  1,  Procedure  used  in  resource-effectiveness  analysis. 
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and  mission  objectives.  In  some  instances  the  system  modfel  may  be  modified 
to  evaluate  an  alternative  solution  to  the  mission  objectives.  This  alternative 
could  require  an  entirely  new  system  model.  System  characteristics  are  cate¬ 
gorized  by  submodels;  each  submodel  symbolizes  a  particular  ty  pe  cf  communi¬ 
cation  equipment.  The  resource  interface  model  (cost  analysis)  assigns  cost 
factors  to  system  characteristics  within  each  submodel.  The  submodel  cost 
expressions  are  then  used  in  the  optimization  program  as  either  criterion  or 
constraint  expressions,  the  use  depending  upon  the  type  of  optimization  being 
perfonned. 

The  optimization  procedure  provides  the  analysis  outputs,  an  analysis  of 
sensitivity  cf  cost  <x  effectiveness  to  changes  in  system  characteristics  in  addi¬ 
tion  to  the  optimized  system  cost  or  effectiveness  configuration. 

In  RESULTS  system  characteristics  are  shown  against  cost  to  indicate  the 
type  cf  trade-off  analysis  possible  fiom  the  method. 

In  the  previously  developed  method,!  the  cost  (resource)  effectiveness 
cf  a  communication  system  included  system  availability  and  system  reliability 
in  addition  to  system  performance  as  measures  of  system  effectiveness.  Avail¬ 
able  time  has  precluded  the  inclusion  of  system  availability'  and  system  relia¬ 
bility'  in  this  report.  Refer  to  appendix  F  for  explanations  of  the  symbols  used 
throughout  the  report. 


Performance 

The  effectiveness  model  which  evaluates  the  performance  of  an  hf  shore- 
to-ship  communication  link  consists  cf  several  submodels.  Each  submodel 
serves  as  an  interface  between  subsystem  characteristics  and  subsystem  cost. 
The  submodels  are: 

1.  Transmitting  terminal 

2.  Transmitting 

3.  Transmitting  antenna  coupler 

4.  Transmitting  antenna 

5.  Receiving  antenna 

6.  Receiving  antenna  coupler 

7.  Receiver 

8.  Receiving  terminal 

Execution  of  the  method  depends  upon  the  ability'  to  associate  a  cost 
factor  with  a  system  performance  factor.  The  procedure  for  determining  cost 
factors  is  described  in  Cost  Analysis.  The  procedure  was  used  to  determine 
the  cost  factors  listed  in  table  1.  These  items  were  shown  to  be  critical  in 
the  establishment  cf  cost  and  performance  factors. 

Subsystem  performance  factors  are  those  major  subsystem  characteristics 
that  directly  influence  the  attainment  of  the  systems  objectives.  The  system 
performance  factors  are  combined  by  analytical  expressions  according  to  com¬ 
munication  theory  to  yield  the  measures  of  system  performance.  The  performance 
as  defined  here  consists  of  information  reliability'  and  information  rate.  System 
cost  factors  modified  b.  system  performance  factors  are  combined  to  give  system 
cost  for  the  predicted  level  of  performance. 
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TABLE  1.  SUBMODEL  COST  AND  PERFORMANCE  FACTORS. 


Performance 

Factor 

Submodel 

cost 

Factor 

Nmnber  of  channels 

Bit  duration 

Type  modulation 

Transmitting 

terminal 

Nmnber  of  channels 
Quantity'  of  units 

Average  power 

Stability 

Transmitter 

Average  power 
Frequency 

Stability 

VSWR 

Transmitting 

antenna 

coupler 

VSWR 

Frequency 

Number  of  inputs 

Gam 

Antenna  pattern 

Antenna  orientation 

Transmitting 

antenna 

Cost  per  antenna 

Gain 

Antenna  pattern 

Antenna  orientation 

Receiving 

antenna 

Cost  per  antenna 

Noise  figure 

Receiving 

antenna 

coupler 

Noise  figure 

Sensitivity 

Stability 

Receiver 

Sensitivity 

Stability 

Frequency 

Quantity  of  units 

Nmnber  of  channels 

Bit  duration 

Type  modulation 

Receiving 

terminal 

Nmnber  of  channels 
Quantity  cf  units 

The  minimum-cost  system  may  be  defined  for  a  specific  level  of  perfor¬ 
mance  by  utilizing  optimization  procedures.  Also,  the  maxhnimi-perfonnance 
sy  stem  may'  be  defined  for  a  fixed  investment. 


PROPAGATION  MEDIUM 

Propagation-prediction  programs  provide  a  method  cf  simulating  analyti¬ 
cally  those  geographical  and  environmental  factors  which  affect  a  system’s 
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effectiveness.  There  are  several  computerized  propagation-prediction  programs 
available  to  the  analyst.  The  capabilities  and  limitations  of  the  programs  vary. 
A  survey  of  propagation-prediction  programs  known  and  available  at  NEL'is 
contained  in  appendix  B.  It  is  this  type  of  computer  program  that  will  be  used 
in  propagation  modeling. 

For  this  analysis,  the  NEL  Fligh-Frequency  Radio  Propagation  Computer 
Program1  was  used  for  modeling  the  propagation  medium.  The  program  calcu¬ 
lates  the  following  characteristics  of  the  propagation  path(s)  as  a  function  of 
the  time  of  day,  month,  sunspot  number,  transmitter  location,  and  receiver 
location. 

1.  Noise  voltage  (receiver) 

2.  Propagation  losses 

3.  Skywave  mode  (number  cf  hops,  reflecting  layer) 

4.  Antenna  gain  as  determined  by  take-off  angle  and  angle  of  arrival 
The  program  also  pennits  various  antenna  ty  pes  and  patterns  to  be  considered 
in  the  calculations. 

All  calculations  in  this  report  were  based  ai  a  shore  transmitter  located 
in  Flonolulu,  and  a  shipboard  receiver  located  at  various  distances  along  a  00 
azimuth  from  Honolulu. 

The  other  characteristics  of  the  communications  link  are: 

Sunspot  number  20 

Month  December 

Hour  0600 


INFORMATION  RELIABILITY 

A  major  objective  in  any  communications  system  is  the  reliable  transfer 
of  information.  Information  reliability  may  be  measured  in  several  ways;  how¬ 
ever,  each  method  is  dependent  upon  the  received  energy  per  bit  and  the  noise 
power  density  at  the  receiver.  The  more  common  measures  of  information 
reliability  are  bit  error  rate  (BER).  character  error  rate,  and  word  error  rate. 

Each  measure  of  information  reliability  is  a  function  of  the  BER,  which  is 
dependent  upon  the  normalized  S/N  ratio  (R). 

The  current  computerized  model  for  information  reliability  does  not  convert 
the  probability'  of  bit  error  (Pe)  to  the  corresponding  normalized  S/N  ratio. 

This  conversion  i  s  made  outside  theprograni  and  i  s  selected  from  the  appro¬ 
priate  equation  listed  in  reference  (1).  Equation  (l)relates  the  S/N  ratio  im¬ 
provement  possible  in  the  detection  model,  than  the  predetection  S/N  ratio 
(RPRE)  to  the  postdetection  S/N  ratio  S0/N0.  The  analysis  of  FSK  detection 
which  follows  is  for  a  nonlinear  receiver  with  a  single  filter  and  linear  discrimi¬ 
nator. 


RPRE  =  c  1-5  x  —h -  x 

0.84(B,7)L  N0 
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where 

VN0=RT0B1 

fm  =  Bj/2  =  3/(4T0) 

B^/^) 

(Bif)L~  B,y  =  2D  +  B  j 


then 


RPRE  =  cl- 5  x 


B/(4Tn)  w  3RTQ 
O.S4(2D  +  3/(2T0))  2(TQ  -  S) 


(1) 


RPRE  is  a  function  of  modulation  type,  receiver  filtering,  pulse  length, 
guard  time,  and  number  of  channels. 

Equation  (2)  equates  RPRE  to  the  propagation  medium  characteristics  and 
performance  parameters  cf  the  system  submodels. 


1°  logl0  (RPRE)  =  10log1QPt  +  30  t  Gt+  G,  t  lOlog 


to 


4r 


(2) 


(1  +  rct) 

-  Lp  -  F  -  C,  -  101og10[l+fMc/s(Sr+S()J+KTB 


where 


F„  =  E„  -  20  logI0  f„c/1  t  65.5  +  10  log10  b 
KTB  =  134  -  10  log1Q  b 

The  following  propagation  factors  are  obtained  as  output  data  limn  the 
NEL  propagation-prediction  program. 6 


Propagation  loss  (Lp) 

Transmit  antenna  gain  (Gf) 

Receive  antenna  gain  (Gr) 

Effective  antenna  noise  (B«) 

Frequency  cf  operation  (l  Mc/s ) 


Tlie  foregoing  propagation  factors  pennit  a  system's  performance  to  be 
analyzed  for  a  specific  geographical  location,  time,  and  range. 


INFORMATION  RATE 

h  the  design  cf  communication  systems,  there  are  certain  trade-offs  that 
can  be  made  between  information  reliability  and  information  rate.  A  computer 
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subroutine  has  been  written  that  enables  these  trade-offs  to  be  taken  into 
account.  Examples  cf  the  trade-offs  possible  are  as  follows: 

The  greater  the  duration  (slow  rate)  cf  the  pulse  element,  the  greater  the 
probability  (high  reliability)  cf  receiving  the  element  correctly.  The  converse 
is  also  true.  Also,  the  more  errors  there  are  in  a  message,  the  less  information 
can  be  transferred  firm  source  to  user. 

When  two  systems  that  use  different  coding  techniques  are  compared,  a 
detailed  analysis  is  required  to  determine  the  rate  at  which  information  is 
transferred.  In  teletype  systems,  the  start  and  stop  bits,  as  well  as  error  detec¬ 
tion  and  correction  bits,  must  be  considered  in  the  determination  of  information 
rate.  When  two  systems  that  use  the  same  coding  technique  are  compared,  it 
is  not  difficult  to  determine  their  relative  information  rates.  However,  coding 
techniques  and  information  rate  must  be  related  to  some  common  factor  for  in¬ 
corporation  within  the  method.  This  common  factor  is  given  by  equation  (3). 
Equation  (3)  states  that  the  channel  information  rate  is  equal  to  the  source  rate 
reduced  by  the  channel  equivocation. 

Irc=  H( x>  ~  Hy(*)  (3) 

lRC  =  Rate  at  which  information  is  transferred 
H(x)=  Source  transmission  rate 
Hy(x)  =  Equivocation 

The  source  information  rate  (Hx)  is  modified  to  consider  both  start  and 
stop  bits  as  well  as  error-detection  bits  to  facilitate  calculations.  That  is,  the 
source  data  rate  is  reduced  by  the  effect  of  start  and  stop  bits  and  error-detection 
bits.  In  the  final  measure  of  information  transferred  by  the  source,  information 
rate  is  further  reduced  ty  the  channel  equivocation.  For  the  binary'  system  the 
channel  information  rate  is 

Irc=t+  T  [f1-IVl0M1-p*)  +IV°g2pe]  (4) 

l0  *0  1  J 

The  overall  system  information  rate  is  directly  proportional  to  the  number  of 
channels  c  and  inversely  proportional  to  the  order  cf  diversity  m  as  given  in 
equation  (5) 


*RT  “  C/m  X  *RC 

Data  Base 


(5) 


One  of  the  problems  associated  with  cost-effectiveness  analysis  is  the 
collection  of  reliable  data.  Standardized  sources  cf  system  characteristics  and 
cost  are  definitely  lacking.  Steps  have  been  taken  to  establish  a  data  storage 
and  retrieval  system  at  NEL  to  support  the  resource-effectiveness  program.  This 
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data  base  will  be  continually  updated  to  provide  a  sliding  historical  data  base. 

The  storage  and  retrieval  program  will  provide  data  fiom  which  cost  factors  can 
be  determined.  In  order  for  cost  effectiveness  to  become  a  continuing  part  of 
die  Navy  procurement  process,  a  complete  data  base  of  Navy  equipment  i  s 
required.  The  best  time  to  acquire  equipment  data  is  in  the  initial  stages  of  a 
system’s  life  cycle.  The  types  cf  data  that  are  required  to  support  resource-effective¬ 
ness  analysis  are  as  follows: 

1  •  Standardized  listing  cf  equipment  characteristics  by  type  of  equipment 

2.  MTBF 

a.  Specification  value 

b.  Predicted  v  alue 

c.  Actual  value 

3.  MTTR 

a.  Specification  value 

b.  Predicted  value 

c.  Actual  value 

4.  Unit  cost 

a.  Quantity  procured 

b.  Date  procured 

c.  Spares 

5.  Training  cost 

6.  Installation  cost 

1.  Personnel  requirements 

8.  Development  costs 

The  data  sources  used  to  support  the  resource-effectiveness  analysis  of 
the  hf  link  contained  in  this  report  are  listed  in  appendix  C  with  type  of  data. 


Cost  Analysis 

The  cost  analysis  applied  here  assigns  a  cost  factor  to  each  specific 
equipment  characteristic  so  that  equipment  costs  can  be  predicted  as  a  function 
of  equipment  characteristics.  A  computerized  statistical  multiple-regression  pro¬ 
gram  is  used  to  determine  each  cost  function  from  historical  cost  data  and  equip¬ 
ment  characteristics.  A  brief  explanation  of  the  theory  and  analytical  process 
involved  in  curve  fitting  via  multiple  regression  is  contained  in  appendix  D# 

Total  equipment  costs  can  be  considered  to  consist  cf  fixed  costs  and 
variable  costs.  Fixed  costs  are  independent  of  the  performance  expressions 
(.METHODS,  equations  2  and  5).  Variable  costs  vary  with  level  of  performance. 
Fixed  costs  here  also  include  performance  factors  that  fail  to  correlate  with  cost. 

Total  cost  =  fixed  cost  t  variable  cost. 

On  the  first  try  at  determining  a  cost  expression  for  a  particular  type  of 
equipment  -  for  example,  a  receiver  -  all  equipment  characteristics  (all  on 
which  there  are  sufficient  data)  are  submitted  to  the  regression  program  at  once. 

If  the  equipment  cost  fails  to  correlate  with  the  pertinent  equipment  character¬ 
istics,  the  equipments  are  further  categorized  by  frequency  range,  type  installa¬ 
tion,  or  other  differences.  Graph  plotting  of  equipment  characteristics  versus 
cost  may  be  employed  to  help  determine  the  analytical  fonn  cf  the  cost  expres¬ 
sion. 
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In  some  instances  the  equipment  characteristics  required  to  evaluate  a 
system’s  effectiveness  may  not  correlate  with  equipment  cost.  Such  pertinent 
equipment  characteristics  as  modulation  types,  number  cf  types  of  modulation, 
transistorized  equipment,  number  of  channels  and  isolation  did  not  correlate  at 
a  significant  level  with  equipment  cost.  It  is  not  implied  that  they  will  not 
correlate,  but  only  that  information  in  extreme  depth  was  not  available.  Also, 
equipment  characteristics  may  correlate  negatively  with  cost  (that  is,  receiver 
cost  decreases  with  a  decrease  in  sensitivity),  hi  lower-cost  equipments 
($3000  or  less)  the  quantity  cf  units  procured  affects  the  unit  cost  significantly. 
In  this  price  area,  as  well  as  with  the  more  expensive  equipments,  the  quantity 
of  units  procured  is  considered  when  this  type  of  information  is  available. 

If  the  data  on  a  particular  type  of  equipment  are  insufficient  to  permit 
curve  fitting,  the  equipment  should  be  treated  as  a  discrete  entry'  in  the  perfor¬ 
mance  and  cost  equations.  This  approach  was  taken  in  the  following  analysis 
with  respect  to  shipboard  and  shore  antennas. 

The  computer  program  used  in  the  curve-fitting  process  provides  several 
statistical  tests  to  evaluate  the  “goodness”  of  the  fitted  curve.  These  tests 
are  the  t  test  and  F  ratio  test  that  are  described  in  appendix  D.  The  multiple- 
correlation  coefficient  and  standard  error  of  estimate  are  also  calculated. 


Optimization  Technique 

One  of  the  main  tools  cf  the  resource-effectiveness  met  iod  is  the  Systems 
Optimization  Program  (SOP).7  Several  minor  modifications  have  been  made  to 
the  SOP,  some  as  adaptations  to  the  current  problem,  and  some  for  compatibility' 
with  NEL  computing  equipment.  The  SOP  minimizes  a  given  function,  called 
the  criterion,  while  satisfying  two  types  of  constraints.  The  constraints  can  be 
in  the  fonn  of  equations  or  bounds  on  the  individual  variables.  These  constraints 
are  always  satisfied  during  the  optimization  procedure. 

hr  connection  with  the  present  evaluations  of  communication  systems,  the 
cost  is  written  as  a  function  cf  system  parameters,  and  this  expression  becomes 
the  criterion  equation  in  the  SOP.  Only  one  constraint  equation  is  used;  in  it 
the  gains  (power,  antenna  gain)  are  balanced  against  the  losses  (path  loss, 
noise,  S/N  ratio).  Some  of  the  variables  are  also  constrained  within  preset 
limits. 

The  roles  of  the  cost  and  gain-loss  equations  can  be  interchanged;  that 
is,  the  cost  can  be  held  less  than  or  equal  to  a  certain  amount.  The  gain-loss 
equation  can  be  used  as  the  criterion,  ty  writing  S/N  ratio  in  tenns  cf  the  gains 
and  losses.  Then  the  SOP  will  maximize  S/N  ratio  by  minimizing  its  negative. 

The  SOP  has  four  major  subroutines,  called  Mode  1,  Mode  2,  Mode  3,  and 
Mode  5.  Mode  1  is  the  most  important,  as  it  executes  the  optimization.  The 
technique  used  is  the  method  of  steepest  descent,  modified  to  work  with  con¬ 
straints  (see  appendix  E*.  The  cost  and  criterion  equations  are  written  as 
functions  of  the  system  parameters.  Some  of  the  parameters  are  variables  and 
may  be  perturbed  in  the  process  cf  minimizing  the  criterion.  These  parameters 
currently  are  transmitter  coupler  VSWR.  transmitter  and  receiver  stability, 
receiver  sensitivity,  transmitter  power,  and  receiver  coupler  VSWR.  A  greater 
number  of  the  system  parameters  are  held  fixed,  but  may  be  changed  for  each 
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ran  of  the  program.  They  include  transmitter  operation  time,  number  of  trans¬ 
mitters  purchased,  path  loss,  and  environmental  noise. 

Mode  2  evaluates  chosen  functions  and  their  derivatives  while  vary  ing 
one  parameter.  It  can  also  plot  these  results,  a  capability  which  allows  the 
analyst  a  convenient  way  to  judge  the  effects  of  indiv  idual  parameters  upon  the 
complete  system. 

Mode  3  evaluates  chosen  functions  while  varying  several  parameters  simul¬ 
taneously. 

Mode  5  is  a  sophisticated  output  routine  that  lists  the  results  of  Mode  1 
in  a  complete  and  readable  fonn.  It  can  also  convert  units  from  those  convenient 
for  calculation  in  Mode  1  to  those  appealing  to  persons  using  the  results. 


Confidence  level  of  Predicted  Performance 


The  performance  equation  contains,  or. is  dependent  upon,  several  para¬ 
meters  for  which  one  cannot  specify  a  “true”  v  alue  but  only  the  most  likely 
value.  Hence,  these  parameters  are  represented  in  the  performance  equations 
ty  their  most  likely,  or  mean,  values.  A  statistical  confidence  factor  is  used 
to  compensate  for  the  effect  of  parameter  uncertainty  upon  system  performance, 
thereby  assuring  a  specified  level  of  performance  with  a  given  degree  of  confi¬ 
dence.  The  confidence  factor  is  included  in  the  performance  expression  (METHODS, 
equation  2)  as  additional  system  loss.  The  confidence  factor  is  determined 
from  system  parameter  uncertainties 8  and  the  desired  lev  el  of  confidence. 

The  uncertainties  are: 

OgiG  =  uncertainty  in  predicting  signal  strength  over  ionospheric  path. 

<?TA  =  noise  variation  about  the  mean. 

aANT  =  uncertainty  in  receiver  antenna  gain  due  to  receiver  antenna 
characteristics. 

o Fa  =  uncertainty  in  mean  value  of  noise. 

°NF  =  uncertainty  in  receiver  noise  figure  value. 

The  total  uncertainty  aj  is 


Z  X  Z  X  Z  1  Z  4.  _  * 

°ANT  +aSIC  +  aTA  +  <7Fa  +aNF 


1/2 


Confidence  factor  (C,)  =  K oT 


Confidence  Interval,  Percent  K 


50 

0. 

90 

1,282 

95 

1.645 

99.9 

3.09 

The  confidence  factor  used  in  these  calculations  were  obtained  fiom  the 
following  system  uncertainties. 

asiG  =  8  dB 
a AN T  ~  ® d B 
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(?TA  =  5  d  B 
o Fa  =  5  dB 
a!SF  =  3  dB 

The  total  system  uncertainty  (oj)  is  found  to  be  13.675  dB. 
dence  factor  for  a  95-percent  confidence  is  then: 


The  confi- 


Cf  =  KaT  =  1.645  x  13.675  dB 


22.495  dB 


RESULTS 


The  results  of  analysis  of  hf-link  effectiveness  are  presented  in  various 
fonns  to  illustrate  the  capability  and  versatility  of  the  method  so  far  developed 
under  this  program.  Performance  and  cost  equations  utilized  in  the  optimiza¬ 
tion  program  are  discussed  with  respect  to  contributing  factors  and  their  inter¬ 
relationships.  Depending  upon  the  fonn  of  the  optimization  process,  the  per¬ 
formance  equation  may  be  either  the  criterion  or  the  constraint  equation.  Cor¬ 
respondingly,  the  cost  equation  will  then  be  the  constraint  or  the  criterion 
equation. 

In  the  optimization  program  a  local  minimum  is  sometimes  found  rather 
dran  the  global  minimum.  In  this  case  the  starting  values  for  the  variable  para¬ 
meters  are  changed  to  determine  if  the  optimization  process  can  locate  a  new 
minimum  for  the  criterion  expression. 

The  optimization  program  uses  26  parameters  (fig.  2),  of  which  only  six 
are  allowed  to  vary'  (variable  parameters).  Other  parameters  could  be  selected 
as  variable  parameters;  however,  die  six  selected  (fig.  2B)  are  believed  to  be 
the  most  significant  with  respect  to  system' cost-performance  trade-offs.  Figure 
2  is  a  ty  pical  SOP  output  page.  It  was  taken  from  a  computer  run  in  which  system 
cost,  parameter  12,  was  used  as  a  constraint  on  system  effectiveness. 
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Fixed  parameters 


PARAM 

NUMBER 

PARAMETER  DESCRIPTION 

fixed 

value 

UNITS 

1 

TRANSMITTING  ANTENNA  GAIN 

-5.VOOOOE 

00 

DEC  1 BELS 

2 

RECEIVING  ANTENNA  GAIN 

-2  .  OoOOOE- 

01 

DECIBELS 

3 

RF  BANDWIDTH 

2.80OOOE 

01 

MEGACYCLES 

4 

CENTER  FREQUENCY 

1.60000E 

01 

MEGACYCLES 

5 

path  loss 

8 • 0  0  0  0  OE 

01 

DECI BELS 

6 

NUMBER  OF  RECEIVERS  PURCHASED 

1 . OOOOOE 

0  1 

DECIBELS 

12 

SYSTEM  COST 

7 • SqoOOF 

04 

DOLLARS 

13 

NUMBER  OF  TRANSMITTER  COUPLER  INPUTS 

1. OOOOOE 

00 

NONE 

15 

NUMBER  OF  CHANNELS 

8. OoOOOE 

00 

NONE 

16 

GUARD  T I  HE 

0 

SECONDS 

17 

BIT  DURATION 

i  .iooooE*- 

02 

seconds 

18 

FREQUENCY  SHIFT 

4.25000E 

01 

CYCLFS  PER  SECOND 

19 

NUMBER  OF  TRANSMITTERS  PURCHASED 

i  .uooooe 

00 

NONE 

20 

ATM.  ,  MANMD.  .  OR  GAL.  NOISE 

1 .  -JoOOOE 

01 

DB  OVER  1  UV/M 

21 

OPERATING  FREQUENCY 

2.40000E 

01 

MEGACYCLES 

22 

1  F  BANDWIDTH 

l.So'OOOE 

00 

RATIO  TO  1KC 

23 

TRANSMITTER  OPERATING  TIME 

7 . 2qOOOE 

02 

HOURS  PER  MONTH 

24 

CONFIDENCE  FACTOR 

2  .24950E 

01 

DECI PELS 

25 

ERROR  RATE 

1  .OoOOOE- 

03 

PER  SECOND 

26 

DIVERSITY 

i .uooooe 

00 

DIVERSITY 

A 

Figure  2.  Typical  printout  for  optimization  program. 
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VARIABLE  PARAMETERS 


PARAM 

NUMBER 

parameter  description 

STARTING 

VALUE 

FINAL 

value 

UNITS 

7 

8 

TRANSMITTER  COUPLER  VSWR 
TRANSMITTER  STABILITY 

2.*0000E  00 
1.00000E-01 

2 ■ 29235E  00 
1.84630E-02 

NONES  PER  1 

9 

receiver  stability 

* .OnOOOE'Ol 

1- 00000E-O2 
MINIMUM  VALUE 

same 

10 

RECEIVER  SENSITIVITY 

5.QQOOOE  00 

3.46897E  00 

MICROVOLTS 

11 

TRANSMITTER  POWER 

1.2?234E  04 

7,725806  03 

WATTS 

14 

RECEIVER  COUPLER  NOISE  FIGURE 

B 

1.00000E  01 

9.966776  00 

DECIBELS 

CO 


Figure  2.  Continued. 
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Submodel  Cost  Prediction 


The  method  of  analysis  required  that  cost  expressions  be  generated  for 
each  submodel  of  the  communications  system.  The  expressions  were  developed 
for  all  but  the  two  antenna  systems,  shipboard  receiving  and  shore  transmitting. 

The  data  available  on  these  types  of  antennas  were  insufficient  to  permit  cost 
expressions  to  be  developed  ty  regression  analysis.  The  costs  and  character¬ 
istics  used  for  the  antennas  were  taken  from  vendor  literature. 

The  cost  expression  for  each  submodel  along  with  the  multiple-correlation 
coefficient  and  standard  error  cf  cost  estimate  (a)  are  included  in  the  summary' 
that  follows.  The  range  for  each  parameter  used  in  the  regression  analysis  is 
also  indicated. 

The  symbol  Q  is  used  to  denote  where  applicable  the  quantity  of  units  to 
be  considered  in  predicting  the  cost  of  a  submodel.  Each  of  the  submodel  cost- 
prediction  expressions  was  generated  by  means  of  the  techniques  described  in 
die  Cost  Analysis  section  of  METHODS.  For  an  explanation  cf  the  symbols 
used  in  the  cost  expression,  see  appendix  F. 

Consider  a  typical  example  of  unit-cost  prediction  using  the  following 
receiver  characteristics  and  the  receiver  cost-prediction  expression  (METHODS, 
equation  3). 

Receiver  Characteristics 

Q  =  30  =  Quantity'  of  units  to  be  procured 
F o  =  16.250  Mc/s  =  Receiver  center  frequency 
Sn  =  1.  =  Receiver  sensitivity 

Sr  =  1  PPM  =  Receiver  stability 

Receiver  cost  ($)  =  t  $5,382,916  -  $2,48269  (30)  -  $659,20691og10(l6.25x  106) 
fixed  cost  variable  cost  variable  cost 

due  to  quantity  due  to  frequency 

-$1,786.0931  [logl0  (10 x  1.5)]  +  $2,016.81387  log10  (100/1) 

variable  cost  variable  cost 

due  to  sensitivity  due  to  stability 

Receiver  Cost  ($)  =  $2,481  when  bought  in  quantities  cf  30. 

The  range  for  each  variable  used  in  the  receiver  cost  analysis  is  as 
follows : 

34  Q  <  2,217  Quantity 

22  £  /<)  <  323,375kc/s  Frequency 

L  <  Sn  <  10^u  Sensitivity 

■  01  <Sr  <  1  PPM  Stability 

Predicted  costs  can  be  obtained  for  the  other  equipments  comprising  the 
system  from  the  appropriate  submodel  equation  in  the  summary  which  follows. 

The  coefficients  assigned  each  cost  factor  are  the  performance  factors  and  are 
determined  by  the  optimization  program.  The  total  hf  system  cost  is  the  sum 
cf  the  individual  submodel  costs. 
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Summan  of  submodel  cost  prediction  equations. 

1.  Terminal  Equipment 

a.  Receive  or  Transmit  Tenninal  Cost  ($) 

=  -843.239356+  1091.535361  (c) 

-69.089276  (Q) 

1  S  Q  ^  25 
1  ^  c  £  16 

Multiple  Correlation  0.99909 
a  =  $635.46589 

b.  Receive  and  Transmit  Tenninal  Cost  ($) 

=  -843.239356+  2183.070722  (c) 

-138.17552  (Q) 

1  ^  Q  ^  5 
1  ^  c  <;  16 

Multiple  Correlation  0.99909 
a  =  $635.46589 

2.  Receiver  Coupler  Cost  (.$)  = 

3014.239578  -  746.668745  (F,)  +  85.11956  (Fc)2  -  3.181748  (Fc)3 
6^  Fc  <,  15 

Multiple  Correlation  0.99005 
a  =  $40.799290 

3.  Receiver  Cost  ($)  =  5,382.916 
-2,486269  (Q) 

-659.2069  log|0fo 
-1786.0931  [log10(lOS„)J 
+  2016.81387  log10(lOO/Srt 
34v  q£  2,217 
22^  fo<;  323, 375kc 
1<S^  10 
0.0l^sr<  1  PPM 
Midtiple  Correlation  0.980429 
a  =  $568.483091 

4.  Transmitter  Cost  ($)  -  -154,942,190818 
+7216.0810  log10p, 

+1,7039107  p, 

+7  1683.2036  (Sbw/f0) 

+9205.5072  log10(100/St) 

o.ou st<;  l ,o ppm 

22  ^  fbu><  28  Mc/s 
\5S  f0  ^  16  Mc/s 
500^  pts  20,000  watts 
Multiple  Correlation  0.997445 
a  =  $1052.209776 

5.  Transmit  Coupler  Cost  $  = 

14,957.591  461 

-2057.4931  logtofo  -349.1224rct 

+1188.842182  ct 

1.1  <:  rct<  2.0:1 

13.15  <;  f0  <;  312.5  Mc/s 
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l<c^  4 

Multiple  Correlation  0.913099 

a  -  $249.653032 
6.  Antennas 

a.  Transmit  Antenna  -  Conical  Monopole  Cost  ($)  =  $6,275 

b.  Receive  Antenna  *  Vertical  Whip  Cost  ($)  =  $300 

The  Effect  of  Primary-Power  Requirements 
Upon  System  Design 

Table  2  illustrates  tire  cost  of  an  hf  communication  link  as  a  function 
of  a  receiver  removed  from  the  transmitter  in  a  fixed  direction  with  range  as  a 
variable  parameter.  The  variables  that  affect  the  system  cost  are  frequency  of 
operation  fMc/s>  external  noise  power  (E,)  available  at  the  antenna,  and  pro¬ 
pagation  loss  (Lp).  The  values  for  these  variables  are  determined  by  the  pro¬ 
pagation-prediction  program  for  each  specific  range.  Also,  the  frequency  of 
operation  selected  is  the  optimum  frequency  for  that  range  and  receiver  location. 
The  S/N  ratio  at  a  specific  receiving  location  is  a  function  of  the  effective 
antenna  noise  figure  and  transmission  path  loss.  The  received  noise  field 
strength  and  operating  frequency  cause  the  effective  antenna  noise  figure  to 
vary  irregularly  with  range.  The  variation  causes  system  price  to  fluctuate 
along  the  selected  path  rather  than  being  monotonic  with  distance.  The  first 
three  line  entries  in  table  2  illustrate  this  situation.  The  path  loss  (L,)  is 
monotonic  with  range,  increasing  from  64.7  dB  at  240  n.m.  to  74.7  dB  at  960 
n.m,  However,  the  effective  noise  at  the  receiver  antenna  (E,)  decreases  from 
14.3 nv  at  240  n.m.  to  3.4 /xu  at  960  n.m.  This  decrease  in  E,  combined  with 
an  increase  in  operating  frequency  more  than  compensates  for  the  10-dB  in¬ 
crease  in  path  loss,  resulting  in  a  less  expensive  system  for  960  n.m.  than  for 
240  n.m. 

In  the  last  two  columns  the  effect  of  primary  power  (or  commercial  power) 
is  shown.  The  primary-power  cost  for  a  5-year  period  was  included  in  the  cost 
(criterion)  equation.  The  equipment  was  considered  to  be  in  operation  720 
hours  per  month.  In  most  cases,  the  effect  of  considering  primary  power  and 
its  associated  cost  was  to  reduce  the  amount  of  the  transmitter  power  output 
required,  causing  the  system  to  seek  additional  gain  from  other  system  para¬ 
meters  at  less  cost  to  the  system. 

The  expression  for  primary-power  cost  uses  the  rate  structure  cf  the 
San  Diego  Gas  &  Electric  Co.,  as  this  information  was  readily  available.  The 
amount  of  primary  power  required  is  dependent  upon  the  efficiency  cf  the  trans¬ 
mitter,  plus  other  items.  The  efficiency  cf  shore  transmitters  was  found  to 
decrease  as  transmitter  poyver  is  increased,  and  this  relationship  was  included 
in  the  expression  for  primary-power  cost.  The  effect  of  transmitter  efficiency 
on  system  cost  becomes  significant  as  the  communication-link  range  is  increased 
from  1,980  n.m.  to  5,040  n.m.  The  communication  link  analyzed  consisted  cf 
eight  digital  channels,  each  operating  with  a  bit  error  rate  (BER)  cf  10' 3  errors 
per  bit.  Hie  link  was  optimized  for  each  range  considered.  The  confidence 
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TABLE  2.  SYSTEM  COST  AND  POWER  REQUIREMENTS. 


Range  t 
n.m. 

Lp, 

dB 

E  n 

fllfc/s 

Widiout  Primary' 

Power  Costs 

With  Prim  ary - 
Power  Costs 

Cost, 

Dollars 

Power, 

Watts 

cost , 
Dollars 

Power, 

Watts 

240 

64.7 

14.3 

4 

46,338 

2,283 

56,830 

1,193 

480 

68.6 

14.8 

4 

49,298 

2,767 

63,599 

1,589 

960 

74.7 

3.4 

20 

35,756 

1,000 

37,663 

250 

1,980 

80.2 

-5.5 

24 

33,321 

500 

36,081 

500 

3,000 

84.1 

-3.4 

20 

37,117 

1,000 

42,114 

500 

4,980 

90.8 

-1.0 

16 

47,292 

1,952 

56,605 

3,139 

5,040 

96.0 

8.2 

8 

166,088 

56,515 

1,400,775 

56,293 

level  associated  with  the  error  rate  is  95  percent.  The  BER  was  determined 
for  noncoherent  FSK  modulation  subjected  to  Rayleigh  fading  and  Gaussian 
noise.  In  addition  to  transmitter  power,  the  optimization  program  varied  other 
system  characteristics.  The  other  system  variables  that  were  permitted  to  vary 
are: 

1  Transmitter  stability 

2  Receiver  stability 

3  Receiver  sensitivity 

4  Receiver  coupler  noise  figure 

5  Transmitter  coupler  VSWR 

Because  of  the  interaction  of  variables  L„  E„  and  fMc/s,  the  worst- 
case  configuration  may  not  occur  at  the  maximum  range  over  which  communica¬ 
tions  are  desired.  Intermediate  ranges,  geographical  locations,  and  tune 
should  all  be  considered  in  determining  the  design  requirements  or  the  adequacy 
of  a  communications  system.  Also  of  importance  in  the  design  cf  a  system  is 
die  transmitter  efficiency  with  its  associated  primary-power  cost. 
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System  Cost  Versus  Range  for  Various  Values 

of  ra 

Figure  3  illustrates  the  effect  of  communication-link  range  on  system 
cost.  The  curves  are  for  effective  antenna  noise  figures  (Fa)  of  40,  50,  arid  60 
dB.  The  link  performance  for  these  calculations  is  constrained  to  an  eight- 
channel  system  operating  with  a  BER  of  10' 3  errors  per  bit  with  a  95-percent 
confidence  factor.  Each  point  plotted  in  figure  3  represents  an  optimized 
(minimum  cost)  system. 


RANGE  (NAUTICAL  MILES) 


Figure  3 .  System  cost  versus  range  for  various  values  of  F0. 
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Specific  communication-link  characteristics  are: 


Link  location 

Modulation 

Sunspot  number 

Month 

Hour 

Transmit  Antenna 
Receive  Antenna 


0  azimuth  from  Honolulu,  Hawaii 
Noncoherent  FSK 
20 

December 

0600 

Conical  Monopole 
Vertical  Whip 


The  curves  illustrate  the  cost  increase  associated  with  a  10-dB  increase 
in  the  effective  antenna  noise  figure.  For  a  2,000'n.m.  link,  an  increase  in  the 
effective  antenna  noise  figure  from  50  dB  to  60  dB  would  require  an  additional 
$21,000  investment  (39  percent  increase)  in  the  system. 


System  Cost  Versus  BER  as  a  Function  of  Fa 


Figure  4  is  a  graph  of  System  Cost  versus  BER  as  a  function  of  Fa 
(effective  antenna  noise  figure)  for  values  of  40,  50,  and  60  dB.  Each  point 
plotted  represents  an  optimized  system  (that  is,  minimum  cost)  evaluated  to 
provide  a  specified  level  of  perfonnance.  All  calculations  are  for  a  fixed  range 
to  eliminate  variations  in  S/N  ratio  due  to  path  loss  and  receiver  location. 

The  characteristics  of  the  link  are  as  follows: 


Range 

Confidence 

Channels 

Modulation 

Location 

Sunspot  number 

Month 

Hour 


3,000  n.m. 

95  percent 
8 

Noncoherent  FSK 
0"  azimuth  (Hawaii) 
20 

December 

0600 


The  graph  illustrates  the  cost  associated  with  designing  a  system  for 
different  effective  antenna  noise  figures.  The  curves  indicate  that-  for  a  very 
slight  increase  in  system  cost  the  BER  can  be  unproved  1km  8  x  10"*  to  5  x  10* 4 
errors  per  bit. 

This  type  of  curve  can  be  used  to  compare  the  BER  improvement  with 
coding  with  that  obtained  from  a  different  system  design. 

Figure  5  illustrates  the  improvement  in  information  reliability  obtainable 
with  an  increase  in  system  cost.  The  calculations  have  been  made  for  path 
losses  cf  70,  80,  and  90  dB.  The  effective  antenna  noise  figure  used  in  the 
optimization  calculations  was  constrained  at  50  dB.  The  optimum  frequency  of 
operation  was  determined  by  the  hf  propagation  -  prediction  program,  and  was 
used  in  each  calculation  for  the  range  involved,  hi  this  type  of  optimization, 
cost  is  a  constraint  and  information  reliability  is  the  criterion. 
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25,000  50,000  75,000  100,000  125,000  150,000  175,000  200,000 

TOTAL  SYSTEM  COST  (DOLLARS) 

Figure  5.  System  cost  versus  Pe  for  various  path  losses. 
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Figure  6  shows  receiver  cost  versus  receiver  center  frequency  for  three 
\alues  of  sensitivity.  The  average  ratio  of  receiver  range  to  center  frequency 
was  1.58.  The  ratio  seems  to  hold  in  most  cases  regardless  of  receiver  center 
frequency. 

Figure  7  illustrates  the  relationship  c£  hf  transmitter  cost  and  average 
transmitter  power  output.  The  curves  have  been  plotted  for  three  levels  of 
transmitter  stability. 


RECEIVER  CENTER  FREQUENCY  (MEGACYCLES) 


Figure  6.  Receiver  cost  versus  center  frequency  for  various  values  of  receiver 
sens  itivity. 
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Sensitivity  Analysis 

The  hf  link  configuration  was  evaluated  to  detennine  the  sensitivity  of 
system  cost  to  each  of  the  six  variable  parameters.  The  system  was  first 
optimized  in  terms  of  cost  (minimum  cost)  to  prov  ide  an  eight-channel  link  with 
each  channel  operating  at  a  bit  error  rate  of  10" 3  errors  per  bit.  The  confidence 
level  over  the  3,00(Vn.m.  link  is  95  percent.  Figures  8  through  13  illustrate 
the  effects  cf  the  variable  parameters  on  system  cost. 

The  optimum  system  cost  was  detennined  to  be  55,530  and  is  indicated  on 
ea  h  parameter  curve.  Figure  9  and  12  indicate  that  system  cost  is  most  sensi¬ 
tive  to  the  characteristics  cf  the  transmitter,  specifically,  transmitter  stability 
and  power.  The  transmitter  cost  analysis  should  be  reviewed  to  verify  the  cost- 
prediction  expressions  before  finalizing  the  analysis. 
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Figure  8.  System  cost  versus  transmitter  antenna  coupler  VSWR. 
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Figure  9.  System  cost  versus  transmitter  stability . 
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Figure  10.  System  cost  versus  receiver  stability. 
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Figure  12.  System  cost  versus  transmitter  power. 
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Figure  13.  System  cost  versus  receiver  antenna  coupler  noise  figure. 


System  Comparison 

The  following  technique  provides  one  mclhod  cf  comparing  two  or  more 
systems  with  multiple  measures  of  effectiveness.  The  effectiveness  cf  each 
system  is  determined  for  a  specific  communications  link.  The  particular  environ¬ 
mental  conditions  used  for  the  comparison  are  a  3,000-n.m.  link  with  an  effective 
antenna  noise  figure  of  50  dB.  In  the  example  selected  each  measure  of  effective¬ 
ness  exceeds  the  system  requirements.  The  technique  is  to  cost  out  the  increased 
effectiveness  of  each  measure  and  subtract  the  increased  effectiveness  cost 
from  the  total  system  cost.  The  lowest  measure  of  effectiveness  in  each  area 
is  used  as  the  basis  for  the  cost  comparison. 

In  this  example,  tire  cost  of  system  1  with  information  reliability  of  2x  10*4 
was  reduced  to  the  cost  of  system  2  w  ith  information  reliability  of  8  x  10~4.  The 
difference  hr  system  cost  is  $15,891,  or  the  excess  performance  cost  of  system 
1  over  system  2  is  $15,891. 

A  similar  comparison  can  be  made  for  system  reliability.  The  result  of 
determining  the  excess  effectivenesscost  is  to  normalize,  the  performance  of 
each  system  for  comparison.  Tire  chart  of  table  3  indicates  that  system  2  is 
the  better  buy  of  the  two  systems. 
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TABLE  3.  SYSTEM  COMPARISON. 


System 

1 

System 

2 

System 

Requirements 

Perfonnance 

Info  Reliability 

2xl0-4 

8xl0-4 

1  x  10"3 

Info  Rate 

Required 

75  baud 

75  baud 

75  baud 

Actual 

76.712 

76.201 

76.05 

System 

1900  hrs 

2300  hrs 

1700  hrs 

Reliability 

(MTBF) 

System 

Availability 

*0.997375 

*0.  9978 

0.99 

System  cost 

480,427 

$65,000 

Excess  Perfonnance 
cost 

$15,891** 

— 

— 

Results  from  $76,427  -  $60,536.  See  figure  4. 
Normalized  to  the  same  level  cf  effectiveness. 


CONCLUSIONS 

1.  The  feasibility  cf  implementing  the  method  developed  in  NEL  report  1323 
has  been  demonstrated. 

2.  There  is  a  definite  need  for  a  standardized  Navy-wide  data  bank  if  cost- 
effectiveness  analysis  is  to  continue  as  a  Navy  design  and  management  tool. 

3.  Detailed  costing  of  equipment  characteristics  such  a  s  kind  of  detection, 
type  of  modulation,  and  use  of  transistors  by  statistical  multiple  regression  may 
not  be  possible,  because  of  other  equipment  characteristics  which  mask  these 
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fine-grain  characteristics  in  the  regression  analysis.  This  costing  may  require 
the  acquisition  of  more  detailed  cost  data  or  the  development  of  new  cost-predic¬ 
tion  techniques. 

4.  The  models  developed  prov  ide  a  base  for  more  detailed  and  comprehensive 
link  analysis. 

5.  The  method  and  techniques  developed  can  be  used  in  the  analysis  and 
evaluation  of  lif  equipments  and  systems.  They  can  also  be  useful  in  the  pre¬ 
paration  of  PTA,  TDP.  and  DAP. 


RECOMMENDATIONS 

1.  Integrate  equipment  reliability  and  maintainability  into  the  existing  method. 

2.  Incorporate  the  effects  of  jamming  and  interference  in  future  refinements 
of  resource-effectiveness  analysis. 

3.  Extend  the  method  to  include  the  communications  system  in  the  operational 
environments . 

4.  Establish  a  centralized  collection  and  distribution  service  concerning 
data  on  Navy  equipments. 

5.  Extend  resource-effectiveness  models  to  encompass  other  frequency  ranges. 
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APPENDIX  A:  A  SINGLE  MEASURE  OF  EFFECTIVENESS 


Several  recent  publications  have  recommended  a  single  measure  of  effec¬ 
tiveness,  or  figure  of  merit,  as  a  means  of  rating  and  comparing  systems.  The 
single  measure  of  effectiveness  is  generally  the  product  of  several  probabilities 
or  rank  designators  that  represent  some  of  the  characteristics  of  the  systems  being 
evaluated.  The  probabilities  can  represent  system  factors,  such  as  reliability, 
system  availability,  or  probability  of  mission  success.  The  ranking  scheme 
involves  rating  these  system  characteristics  on  a  scale  of  a  to  b  (b>  a).  The 
purpose  of  this  appendix  is  to  demonstrate  some  of  the  more  basic  limitations 
associated  with  single  measures  of  effectiveness. 

To  detennine  whether  a  single  measure  of  effectiveness  is  appropriate,  \vc 
must  first  look  to  the  purpose  of  a  resource-effectiveness  analysis.  An  analysis 
of  system  effectiveness  is  designed  to  present  all  significant  alternatives  in 
system  configuration  with  their  inherent  ramifications  of  resource  requirements 
and  mission  fulfillment  in  perspective.  Where  a  single  measure  of  system  effec¬ 
tiveness  is  used,  it  is  difficult,  if  not  impossible,  to  relate  trade-offs  to  the 
overall  system  objective(s),  particularly  in  those  cases  in  which  the'system 
factors  are  weighted  on  some  arbitrary7  basis.  Large  systems  with  multiple 
objectives  are  definitely  more  suited  to  the  use  of  multiple  measures  of  effec¬ 
tiveness  than  to  the  use  of  a  single  measure.  Decision  makers  quite  frequently 
request  a  single  measure  of  effectiveness,  as  it  facilitates  decision  making  and 
decreases  the  administrative  problems  associated  with  it.  A  resource-effective¬ 
ness  analysis  should  present  an  unbiased  array  of  possible  trade-offs.  It  should 
not  force  a  specific  decision  through  previous  decisions  in  the  course  of  analysis 
as  a  single  measure  of  effectiveness  would. 

The  case  in  which  a  system’s  major  characteristics  are  ranked  horn  1  to 
9  and  then  combined  as  a  product  into  a  single  measure  of  effectiveness  is 
examined  in  detail  in  the  following  paragraphs. 

Assume  the  situation  in  which  system  effectiveness  is  the  product  of  four 
system  characteristics  a,  /3.  y.  and  8 
or  effectiveness  =o/9y5 
where 


1  <  9 

1  </3^9 
1  <  y^9 
1  <  5^9 


la,  /3,'y,  8  are  integers. 


Figure  A1  graphically  illustrates  the  cumulative  density7  of  available  pro¬ 
ducts.  There  are  6,561  possible  different  ordered  combinations  cf  a,  P,  y,  and  8 
yielding  225  different  products.  Consider  the  situation  in  which  a  system  rating 
is  3024.  There  are  24  different  system  configurations  defined  by  the  rating  system 
that  will  provide  the  product  of  3024.  A  system  w  ith  effectiveness  parameters 
9*8 *6 *7  is  different  from  a  system  with  parameters  6*7*9*8-  However,  whatever 
difference  the  systems  have  is  masked  by  the  single  measure  of  effectiveness. 

If  the  effectiveness  rating  is  144,  there  are  132  possible  different  system  con¬ 
figurations. 
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Figure  A1 .  Cumulative  density  of  products  (single  measure  of  effectiveness ). 
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Table  A1  indicates  the  number  of  ordered  combinations  that  give  a  particular 
product.  An  ordered  combination  would  consider  the  permutations  possible  with  a 
specific  combination  of  numbers. 

TABLE  A1  .  4-NUMBER  ORDERED  COMBINATIONS  AND  THEIR  PRODUCTS 
(!<;  each  number  ^  9)  (Continued  through  page  E-42) 


Product 

Niunber  Ordered 
Combinations 
Giving  Product 

Number  Ordered 
Combinations 
Product  Giving  Product 

Number  Ordered 
Combinations 
Product  Giving  Product 

1 

1 

50 

12 

■■ 

84 

2 

4 

54 

52 

mm. 

12 

3 

4 

56 

40 

wEam 

84 

4 

10 

60 

60 

SU'-i 

28 

5 

4 

63 

24 

mm 

88 

6 

16 

64 

44 

196 

18 

7 

7 

70 

24 

200 

24 

8 

20 

72 

112 

210 

48 

9 

10 

75 

12 

216 

116 

10 

12 

80 

48 

224 

48 

12 

36 

81 

19 

225 

18 

14 

12 

84 

60 

240 

84 

15 

12 

90 

60 

243 

16 

16 

31 

96 

96 

245 

12 

18 

36 

98 

12 

250 

4 

20 

24 

Bl 

18 

252 

84 

21 

12 

m 

24 

2% 

31 

24 

64 

■K*.: 

88 

270 

60 

25 

6 

112 

48 

280 

48 

27 

16 

120 

84 

288 

120 

28 

24 

4 

294 

24 

30 

36 

60 

300 

24 

32 

40 

40 

315 

36 

35 

12 

28 

320 

40 

36 

72 

36 

324 

72 

40 

40 

144 

132 

336 

84 

42 

36 

147 

12 

343 

4 

45 

24 

150 

24 

350 

12 

48 

88 

160 

48 

360 

96 

49 

6 

162 

52 

375 

4 
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i 

TABLE  Al.  (Continued). 


Number  Ordered 
Combinations 
Product  Giving  Product 

Number  Ordered 
Combinations 
Product  Giving  Product 

Product 

Number  Ordered 
Combinations 
Giving  Product 

378 

60 

756 

60 

m 

4 

384 

64 

768 

36 

m 

12 

392 

24 

784 

18 

36 

400 

18 

800 

12 

mm 

16 

405 

24 

810 

36 

■9 

12 

48 

840 

48 

52 

432 

112 

864 

76 

16 

441 

18 

875 

4 

1568 

12 

448 

40 

882 

24 

1575 

12 

450 

24 

896 

24 

1600 

6 

480 

60 

900 

18 

mm 

24 

486 

36 

945 

24 

■n 

24 

490 

12 

960 

36 

1701 

12 

500 

4 

972 

3.6 

mum 

4 

504 

% 

980 

12 

m 

40 

512 

20 

4 

18 

525 

12 

72 

WSm 

12 

540 

60 

10 

12 

560 

36 

4 

24 

'567 

24 

1050 

12 

12 

576 

88 

1944 

28 

588 

24 

1960 

12 

600 

24 

2016 

36 

625 

1 

1134 

36 

2025 

6 

630 

48 

1152 

48 

2048 

4 

640 

24 

mm 

24 

2058 

4 

648 

76 

mm . 

12 

2106 

24 

672 

60 

m  mi 

12 

2187 

4 

675 

12 

6 

2205 

12 

686 

4 

Ifl 

36 

2240 

12 

700 

12 

1280 

12 

2268 

24 

720 

72 

12% 

55 

2304 

18 

729 

10 

1323 

12 

2352 

12 

735 

12 

1344 

36 

2401 

1 

750 

4 

1350 

12 

2430 

12 
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TABLE  A 1 .  (Continued). 


Product 

Nmnber  Ordered 
Combinations 
Giving  Product 

Nmnber  Ordered 
Combinations 
Product  Giving  Product 

Nmnber  Ordered 
Combination  s 
Product  Giving  Product 

2520 

24 

3072 

4 

3969 

6 

2560 

4 

3087 

4 

4032 

12 

2592 

24 

3136 

6 

4096 

1 

2646 

12 

3240 

12 

4374 

4 

2688 

12 

3402 

12 

4536 

12 

2744 

4 

3456 

12 

4608 

4 

2835 

12 

3528 

12 

5103 

4 

2880 

12 

3584 

4 

5184 

6 

2916 

10 

3645 

4 

5832 

4 

3024 

24 

3888 

12 

6561 

1 
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APPENDIX  B  :  SURVEY  OF  PROPAGATION- 
PREDICTION  PROGRAMS* 


Introduction 


Use  of  digital  computers  for  propagation  calculations  has  grown  with  the 
computer  development.  Thus  a  step-at-a-time  adyance  in  this  use  has  occurred 
with  initial  applications  in  the  hf  spectral  region.  No  comprehensive  programs 
adapted  to  the  entire  spectrum  exist.  Various  groups  have  produced  programs 
restricted  to  use  in  appropriate  spectral  regions. 

In  the  spectral  range  fiom  vlf  to  microwaves  the  most  intense  effort  toward 
computer  solutions  has  been  made  in  the  hf  and  vlf  regions.  Nearly  every  hf 
solution  in  existence  can  trace  its  origin  to  the  procedures  developed  at  Central 
Radio  Propagation  Laboratory  (CRPL).  The  programs  provide  estimates  of 
maximum  usable  frequencies  (MUF),  field  strength,  signal-to-noise  ratio,  and  hop 
structure.  The  vlf  programs  depend  generally  upon  solutions  of  the  modal  equa¬ 
tions  for  the  earth'ionosphere  wave  guide. 

Less  effort  has  been  devoted  to  generating  programs  for  frequencies  above 
hf,  mostly  because  adequate  solutions  for  system  design  are  available  rather 
directly  by  non-machine  procedures.  A  general  exception  to  this  statement  are 
the  ray-trace  programs,  which  have  been  adapted  to  machine  programs  extensively. 
These  programs  provide  a  picture  of  the  energy  distribution  on  the  space  illumi¬ 
nated  by  the  antenna.  Programs  of  this  kind  exist  for  hf  ionospheric  scatter  fiom 
30  to  100  megacycles,  and  for  tropospheric  ducting  in  the  microwave  region. 

Most  provide  some  kind  of  estimate  of  signal  loss  as  well  as  the  primary'  ray- 
trajectory  output. 

Propagation-prediction  programs  have  been  generated  at  the  following 
facilities: 

1.  ITS  A,  ESSA,  U.S.  D.O.C.  (CRPL) 

2.  NRL 

3.  Stanford  University 

4.  Stanford  Research  Institute 

5.  DECO  Electronics,  Inc. 

6.  Collins  Radio  Corporation 

7.  Raytheon  Corporation 

8.  NEL 

9.  AVCO  Corporation 

10.  RCA 

11.  USRPA,  Ft.  Monmouth 

12.  DRTE,  Canada 

13.  Radio  Research  Laboratories,  Tokyo,  Japan 


"The  survey  of  propagation-prediction  programs  was  generated  specifically  for  this  problem 
by  personnel  of  the  NEL  Radio  Physics  Division  under  the  direction  of  C.  H.  Freres. 
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14.  Radio  Research  Station.  Slough,  England 

15.  Ion  Prediction  Service.  Sydney,  Australia 

In  addition  both  DCA  and  NAVCOSSACT  employ  a  wide  range  of  computer 
prediction  programs. 

Brief  summaries  of  programs  about  which  NEL’s  Radio  Physics  Division 
has  more  than  superficial  knowledge  follow  .  They  are  intended  to  do  no  more 
than  indicate  the  scope  and  intended  use  of  the  programs.  A  program-characteris 
tic  matrix  that  provides  some  comparison  detail  is  appended. 


NEL  High-Frequency  Propagation-Prediction 
Computer  Program 

Radio  system  parameters  are  combined  with  geophysical  and  ionospheric 
characteristics  to  predict  the  performance  of  high-frequency  sky-wave  communi¬ 
cation  circuits.  The  program  computes  Maximum  Usable  Frequencies  (MUF), 
probable  modes  of  propagation,  E  layer  MUF  and  cutoff  frequencies,  angles  of 
arrival,  ground  losses,  total  losses,  field  strength,  antenna  gains,  absorption 
losses,  signal  strength,  noise  strength,  and  signal-to-noise  ratios.  In  contrast 
to  the  CRPL  program,  the  program  utilizes  hf  characteristic  charts  for  critical 
frequencies  and  atmospheric  noise  (CCIR  report  65).  The  solution  i  s  divided 
into  two  parts  -  an  estimation  of  the  field  strength  independent  of  equipment 
parameters,  and  an  estimation  of  signal-to-noise  ratio  using  antenna  gains.  The 
program  was  written  for  the  CDC  1604  computer  and  is  in  NELIAC  5m.  There 
is  an  output  for  every  operational  mode,  whereas  the  CRPL  outputs  only  for  the 
optimum  mode.  Computer  tune  is  less  than  for  the  CRPL  program. 


AVCO  Polar  HF  Prediction  Program 

This  program  determines  the  highest  and  lowest  frequencies  available 
between  two  particular  stations  as  a  function  of  time,  and  the  geophysical  and 
ionospheric  parameters.  Propagation  losses  are  detennined  for  specific  fre¬ 
quencies  within  this  calculated  range.  The  calculation  is  extended  to  ionos- 
pherically  disturbed  conditions,  but  all  calculations  are  valid  only  for  high 
sunspot  number.  The  computer  program  is  written  in  standard  Fortran  and  in¬ 
cludes  19  modes  of  propagation.  Frequencies  fiom  3  to  30  Mc/s  in  intervals  of 
3  Mc/s  are  considered.  The  program  prints  mode,  transmission  angle,  and 
available  frequency  range.  In  addition,  it  prints  space,  absorption,  sporadic  E, 
and  total  losses  for  each  test  frequency  within  the  available  frequency  range. 
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Radio  system  parameters  are  combined  with  geophysical  and  ionospheric 
characteristics  to  predict  the  performance  of  high-frequency  sky-wave  communi¬ 
cation  circuits.  The  program  computes  maximum  usable  frequencies,  optimum 
traffic  frequencies,  lowest  useful  frequencies,  probable  modes  of  propagation, 
angles  cf  arrival,  circuit  reliability,  system  loss,  available  signal-to-noise 
ratios,  and  field  strength.  Numerical  representation  is  used  for  all  parameters 
not  expressed  in  closed  mathematical  form,  such  as  world  maps  of  critical  fre¬ 
quency  and  atmospheric  noise.  The  solution  of  the  problem  is  divided  into  two 
parts  -  an  estimation  of  the  available  signal,  and  an  estimation  of  the  required 
signal.  The  program  was  initially  written  for  EM  7090-class  computers  and  was 
translated  at  NEL  to  Fortran  63  for  the  CDC  1604.  At  a  later  date,  NEL  added 
CCIR  report  322  noise  data  to  the  program.  In  the  322  version  the  calculations  of 
circuit  reliability  cannot  be  made. 


Collins  Radio  HF  Program 

The  program  is  similar  to  that  used  by  NBS  and  yields  comparable  data. 

The  differences  are  in  the  calculation  of  Lowest  Usable  Frequency  (LUF)  and 
auroral  absorption.  At  present,  the  median  noise  levels  are  used  to  calculate  the 
frequency  that  satisfies  the  loss  equation;  this  frequency  is  called  the  LUF.  In 
the  NBS  program  for  LUF,  the  loss  equation  is  solved  by'  trial  and  error  for  a 
reliability  of  90  percent,  taking  into  account  changes  of  mode  with  a  two-dimen¬ 
sional  antenna  gain  function.  In  the  NBS  program,  the  additional  loss  is  deter¬ 
mined  by  the  F2-layer  control  point  location,  hi  the  Collins  method,  the  average 
of  two  absorption  indices  is  calculated  fkm  geomagnetic  coordinates  of  the  rays. 

Input  insertion  of  links  is  unique.  As  many  as  150  stations  are  permitted 
with  as  many  as  200  combinations  of  stations  as  links. 


Canada  DRTE  HF  Program 

This  program  computes  the  maximum  usable  frequency  (MU  F)  and  lowest 
usable  frequency  due  to  E  layer  cut-off  of  the  F  layer  for  a  given  mode  or 
modes.  The  basic  ionospheric  data  used  for  the  prediction  of  the  F2  and  E 
layer  MUF  are  obtained  from  DRTE’s  manual  prediction  system. 


VLF  Program  (Pappert]  -  NELIAC  MOD  7  (1604) 

The  program  solves  the  earth-ionosphere  wave-guide  mode  equation.  In 
general,  Budden’s  formalism  is  employed.  Solutions  to  Stokes’  equation  in 
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tenns  of  third-order  modified  Hankel  functions  are  employed  to  determine  the 
reflection  matrix  solution  to  the  problem  of  the  three-layer  boundary  value. 
Gossard’s  solution  for  the  upgoing  wave  is  incorporated  in  the  Pappert  solution. 

Required  program  inputs  are  the  initial  admittance  matrix  (Gossard-Smith 
program),  profile  increment,  integration  limits,  path  data,  geophysical  data, 
and  control  parameters. 

The  program  output  lists  the  characteristic  mode  angles,  phase  velocity, 
attenuation,  excitation  factor,  and  modulus  of  the  polarization  vector. 


VLF  Program  (Stnith/G  ossardj 

A  full-wave  solution  is  made  for  electromagnetic  propagation  in  a  con¬ 
tinuous  ionosphere  with  arbitrary  parameters  (electron  density  and  collision 
frequency).  Budden’s  solution  is  the  basis. 

Inputs  are  initial  conditions  at  high  altitude,  height,  complex  angle  of 
incidence,  propagation  angle,  dip  angle  and  magnetic  field  strength,  frequency, 
collision  frequency,  and  vertical  profile  of  electron  density'. 

The  basic  program  output  is  the  reflection  coefficient  matrix  expressed  in 
polar  coordinates  as  a  function  of  height. 

The  program  employs  a  procedure  for  finding  an  appropriate  initial  value 
for  the  reflection  coefficient  prior  to  performing  the  real  ionosphere  integration. 


DECO  Program 

This  program  predicts  the  mean  intensity'  of  atmospheric  noise  for  any 
frequency  for  which  a  wave-guide  model  cf  earth-ionosphere  is  acceptable.  The 
wave-guide  mode  equation  is  used.  The  program  assumes  that  the  mean  noise 
intensity  at  a  receiver  may  be  simulated  by  properly  combining  fields  produced 
by  a  number  of  transmitters  that  replace  the  actual  thunderstorm  sources. 

The  program  output  is  the  summed  field  intensities  for  given  locations  and 
times.  With  an  appropriate  plotter  worldwide  contour  maps  of  noise  intensities 
can  be  produced. 


DECO  Program  (NEL  Variation) 

The  basic  DECO  program  will  be  modified  to  provide  predictions  for  the 
phase  and  amplitude  cf  given  transmitters  for  any  location  on  the  earth. 
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Kift-fooks  Ray  Trace  (Stanford/NEL  Version) 

This  program  employs  a  simplified  approximate  procedure  for  tracing  rays 
between  a  terminal  pair.  It  provides  an  assessment  of  possible  hop  stractures, 
using  a  zeroing-in  procedure  to  reject  all  rays  not  within  ±100  km  of  the  receiving 
point.  The  ionospheric  model  is  keyed  to  the  CRPL  predictions,  but  represents 
the  layers  as  parabolic  and  concentric.  The  magnetic  field  effects  are  not 
included. 

The  output  includes  the  path  modes,  frequency,  take-off  angles,  great- 
circle  path  length,  travel  tune  along  the  ray  path,  ionospheric  absorption  loss 
(only)  along  route,  and  the  maximum  usable  frequency. 

The  inputs  required  are  the  tenninal  coordinates,  the  appropriate  CRPL 
predicted  ionosphere  or  a  measured  set  of  vertical  soundings,  declination  of  sun, 
frequencies,  range  take-off  angle  aperture,  and  time.  E,  layers  may  be  included 
“after  the  fact.” 

Separate  auxiliary'  programs  provide  for  tape-loading  the  CRPL  ionosphere, 
and  for  plotting  the  outputs  on  the  1 60-A  printer. 


Ray  Trace  Program  (Sheddy] 

The  Haselgrove  equations  are  used  to  trace  ray  paths  in  three  dimensions 
in  a  model  ionosphere.  The  ionospheric  model  is  a  three-dimensional  combina¬ 
tion  of  parabolic  layers  and  CRPL  (Gallet- Jones)  world  maps.  It  traces  both 
ordinary'  and  extraordinary'  rays.  No  special  provision  has  been  made  to  use 
values  of  frequency  below  the  gyro  frequency. 

Input  parameters  are  tenninal  coordinates  or  azimuth  and  maximum  dis¬ 
tance,  elevation  angle,  frequency,  year,  month,  and  hour. 

Outputs  include  great-circle  path  length,  ray  angle,  ray  height,  and 
geographic  coordinates  at  each  computed  point. 

The  program  pennits  ready  substitution  of  alternate  ionospheric  models. 
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Characieristics/Source 

NEL/hf 

AVCO/hf 

CRPL/hf 

Collins/hf 

Canada 

DRTE/hf 

lonoscatter 

SRI 

SR) 

Trop 

Scatter 

NEL/vlf 

(Pappert) 

NEL/vlf 

(Smith- 

Gossard) 

DECO/vlf 

NEL/Standard 

Kift/Fooks 

Ray  trace 

NEL 

$hed(tycc 

General 

Computer 

CDC  1604 

IBM  7094 

IBM  7090 
CDC  1604 

Unknown 

|  evl  650 

IBM  7090 

IBM  7090 

CDC  1604 

CDC  1604 

CDC  3600 

CDC  1604 

CDC  1604 

Language 

NELIAC 

5M 

FORTRAN 

FORTRAN 

Unknown 

Unknown 

FORTRAN 

FORTRAN 

NELIAC 

7* 

NELIAC 

FORTRAN 

FORTRAN 

FORTRAN 

Geographic 

Limitations 

None 

None 

None 

None 

Northern 

Latitudes 

None 

None 

None 

None 

None 

None 

Ionospheric 

Model 

Graphical 
Represen¬ 
tation  cf 
ionospheric 
date 

Numeric 

map 

Numeric 

map 

Same  as 
CRPL 

DRTE/RPL 
ion  para, 
and  mono-  | 

Same  as 
NEL/hf 

Numeric 

map 

Numeric 

map 

Availability 

In-house 

Unknown 

In-house 

Unknown 

Unknown 

In-house 

In-house 

In-house 

In-house 

In-house 

Speed 

480  modes 
minute 

400  modes 
minute 

Unknown 

3  min /mo.  , 
cf  MUF 
LUF  data 

Special  features 

Detailed 
output 
and  14 
types  of 
antennas 

Consider 

disturbed 

ionosphere 

3  antenna 
types 

Auroral 

sont' 

absorption 
and  6 
antenna 
types 

Include 

IF  layer 

Includes 

jamming 

Considers 

jamming 

Fteauencies 

3-30  Mr/s 

3-30  Mc/s 

3-30  Mc/s 

3-30  Mc/s 

3-30  Mc/s 

10- TO  kcA 

10-30  kc/s 

330  Mc/s 

Inputs 

Year 

No 

Yes 

No 

No 

No 

1 

Yes 

Month 

Yes 

Yes 

Yes 

Yes 

Yes 

res 

Yes 

Sunspot  number 

Yes 

Yes 

Yes 

i  Yes 

Yes 

Yes 

KHHilWii 

Jammer  coordinates 

Yes 

Transmitter 

coordinates 

Yes 

Yes 

Yes 

fes 

Yes 

Yes 

Transmitter  altitude 

Yes 

Receiver  coo  rdi  na  tes 

Yes 

Yes 

Yes 

Yes 

Yes 

fes 

Yes 

Yes 

Yes 

Effective  receiver 
temperature 

Yes 

Path  length 

No 

No 

No 

No 

Opt 

Receiver  altitude 

Yes 

I  Opt 

Azimuth 

No 

No 

N> 

No 

Surface  refractivity 
at  R&T 

Yes 

Power 

Yes 

No 

Yes 

Yes 

(cs 

Yes 

Frequencies 

Yes 

No 

Yes 

No 

r'es 

Yes 

Antenna  pattern 

No 

No 

opt. 

Yes 

fes 

Yes 

No 
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Chaiacteristics/Source 

NEL/hf 

AVCO/hf 

CRPL/hf 

Collins  /hf 

Canada 

DRTE/hf 

tonoscatter 

SRI 

SRI 

Trop 

Scatter 

NEL/vlf 

(Pappert) 

NEL/vlf 

(Smith- 

Gossard) 

DECO/vlf 

NEL/SUndaid 

Kift/  Foots 
Raytrace 

NEL 

Sheddv 

Raytrace 

Inputs  (Continued) 

Antenna  bear  in* 

Opt. 

No 

Opt. 

No 

Yes 

No 

Launch  elevation 
angle 

Yes 

Man  made  noise  1  Om.  1  No  1  Opt. 

Mo 

Universal  tune 

Yes 

Yes 

NO 

Yes 

Yes 

Yes 

Yes 

Day  cf  month 

No 

Yes 

No 

No 

Davcf  year 

No 

Yes 

Nb 

No 

Magnetic  index,  Kp 

NO 

Yes 

No 

No 

Ionospheric  storm 

parameters 

No 

Yes 

No 

No 

Required  S/N 

No 

No 

Yes 

Yes 

BOW 

Yes 

No 

No 

Yes 

Yes 

Outputs 

Absorption  losses 

Yes 

Yes 

No 

Yes 

Space  losses 

Yes 

Yes 

No 

Sporadic  E  losses 

No 

Yes 

No 

Ground  losses 

Yes 

No 

No 

Field  strength 

Yes 

No 

Yes 

Reflection  coeffic¬ 
ients 

Yes 

Signal  strength 

Yes 

No 

No 

Yes 

- - i 

Yes 

Antenna  gains 

Yes 

N> 

Yes 

Yes 

Noise 

Yes 

No 

No 

Yes 

Delay  time 

No 

No 

N> 

Yes 

Yes 

Effytive  height 

Yes 

N> 

t-b 

Transmission  angle 

Yes 

Yes 

Yes 

Mf&.  transmission 
angle 

Yes 

N> 

No 

Modes 

Yes 

No 

Yes 

Yes 

MUF-HAF 

Yes 

Yes 

Yes 

Yes 

Yes 

Service  Probability, 
reliability 

No 

No 

Opt 

LUF 

no  : 

Yes 

Opt 

Yes 

Yes 

FOT 

No 

No 

Opt 

Yes 

Signal-to- noise 
ratio 

Yes 

No 

Yes 

Yes 

Auroral  Zone 
Absorption 

No 

Yes 

No 
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M 

I 

Ln 


Character  istics/Source 


Outputs  (Continued) 

Equatorial  Zone 
Absorption _ 

EMF _ 

Path  identification 
Path  length 
Time,  month 

Year  or  sunspot 
number 

Power 

Frequencies  opera¬ 
ting _ 

Reflection  Point 

Coord. 

Layer  height 
Raj  height 
Total  losses _ 

BDW 

Required  S/N 

Turbulent  signal 

component 

Meteor  signal 
component 


SRI  NEL/vlf  NEL/Standard  NEL 

Canada  Ionoscatter  Trop  NEL/vl(  (Smith-  Kift/Fooks  Shoddy 

Collins/hf  DRTE/hf  SRI  Scatter  (Pappert)  Gossard)  DECO/vlt  Raytrace  Raytrace 


Yes 
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APPENDIX  C:  DATA  TYPE  AND  SOURCE 


Data  Type  Data  Source 


Average  Prices  Bureau  of  Ships  NavShips  92.563(B), 

of  Navy  Electronic  Equipment  Index  to  Bureau  of  Ships  Controlled  Elec¬ 

tronics  Equipment  (F  Cognizance).  15 
March  1963 

Bureau  of  Ships  NavShips  900.123(D), 
Nomenclature  Assigned  to  Naval  Elec¬ 
tronic  Equipment,  CONFIDENTIAL, 

August  1%3 

Equipment  Characteristics  Bureau  of  Ships  Nav  Ships  94,200.01, 

and  some  price  information  Directory  of  Communication  Equipment, 

SECRET,  April  1964 

Bureau  of  Ships  NavShips  94,200.0100, 
Directory  of  Classified  Electronics  Major 
Units,  CONFIDENTIAL  ,  January  1964 

Bureau  of  Ships  NavShips  94,200.1, 

Section  1 ,  Directory  of  Communication 
Equipment,  n.d. 

Bureau  cf  Ships  NavShips  94,200.1,  Section 
2,  Directory  of  Communication  Equipment, 
n.d. 

Bureau  of  Ships  NavShips  94,200.1,  Sec¬ 
tion  3,  Directory  of  Communication  Equip¬ 
ment,  n.d. 

Bureau  of  Ships  NavShips  94,200.1,  Sec¬ 
tion  4,  Directory  of  Communication  Equip- 
ment.  n.d. 


Bureau  of  Ships  NavShips  94,200.1,  Sec¬ 
tion  5,  Directory  of  Communication  Equip¬ 
ment,  n.d. 


Bureau  of  Ships  NavShips  94,200.1, 
Section  6 ,  Directory  of  Communication 
Equipment,  n.d. 

Bureau  of  Ships  NavShips  94,200.1, 
Section  7,  Directory  cf  Communication 
Equipment,  n.d. 


General  Characteristics 

cf  AN  ty  pe  Electronic  Equipment 

by  AN  designation 
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Bureau  of  Ships  NavShips  94,200.1, 
Section  8,  Directory  of  Communication 
Equipment,  n.d. 

Bureau  of  Ships  NavShips  94,200.1, 
Section  9,  Directors  of  Communication 
Equipment,  n.d. 

Equipment  Characteristics  Ships  Specifications 

and  specification  MTBF  value 

Equipment  Cost,  F  Cognizance 

Quantity  and  Date  of  procurement  Material  Control  Branch 

(Code  6627) 

Electronic  Equipment  Fleet  Electronic  Effectiveness  Branch 

MTBF  and  MTTR  values  (Code  6678) 
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Regression  analysis  presupposes  that  some  relationship  exists  between  a 
dependent  variable  Y  and  one  or  more  independent  variables  X  j ,  X2,  X3,  —  X„. 
The  simplest  case  is  approximated  b.  the  linear  equation  of  the  fam 


n 

Y  =  fio  +  'l  Pixi  +  e' 

i-  1 

/3,  are  the  parameters  in  a  one-dimensional  space  generated  ty  the  regression 
plane.  The  quantity  e'  represents  tlie  random  or  simple  error  in  the  variation  of 
Y not  accounted  for  in  the  regression  plane.  General  transformations  such  as 
X  x '  =  1  / Xj  or  X2'  -  X22  will  yield  an  equivalent  fcmi  of  the  foregoing  equation. 

The  true  values  of  constants  /3q,  ,  /32.  ~  ~  '  fin  can  never  be  determined. 

However,  estimates  of  these  constants  can  be  obtained  from  m  observations  cf 
Y  and  corresponding  X;  values.  A  simple  wav  cf  writing  the  m  observations  is 
in  tire  lam  of  a  table  a  s  shown  below: 


*1 

x2 - 

1 

X 

3 

Vi 

all 

a  12  ‘ 

'  ■  ■  *  aln 

Y2 

a21 

a22  '  ‘ ‘  ' 

a2n 

Ym 

am  1 

am  2'  '  '  ’ 

•  -  -  amn 

The  linear  estimating  equation  then  has  the  fam 
n 

Y  =  a0  +  ^  a,X,  +  < 

i-  l 


where 


a  ■  is  the  regression  coefficient  and  is  an  estimate  cf  the  true  but  unknown 
coefficient  /3, 

e  is  the  residual  of  the  true  Y  about  the  regression  plane 
X ;  is  the  independent  variable 

Several  assumptions  are  made  about  the  independent  and  dependent  variables 
that  permit  significance  tests  and  confidence  interval  estimates  to  be  made. 

These  assumptions  also  lend  themselves  to  the  least-squares  method  cf  esti¬ 
mating  the  value  of  a;. 

These  assumptions  are 

1.  AUX;  ’s  are  fixed  variables  (that  is,  there  are  no  probability  distributions). 
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2.  The  V’s  are  all  normally  and  independently  distributed  about  the  mean 

n 

(ao  t  2  a(.  X[  )\v ith  \ariance  a2  ■ 
i  =  1 

The  least-squares  method  minimizes  the  sum  of  squares  of  de\iation  (G)  fern  the 
estimated  regression  plane. 


m 


Graphical  procedures  are  used  to  determine  the  fcrni  and  transformations  re¬ 
quired  in  the  linear  regression  equation. 

Multiple-regression  analysis  makes  use  of  several  statistical  tests  to 
detennine  the  significance  of  coefficients  and  equations.  The  F'test  determines 
whether  the  Icon  of  the  equation  is  statistically  significant.  The  t-test  checks 
the  significance  of  the  partial-regression  coefficients.  The  multiple-correlation 
coefficient  R  gives  the  degree  of  correlation  between  the  dependent  variable 
Y  and  the  independent  variables  Xj,  X^<  X3  —Xn.  A  more  detailed  discussion 
of  these  tests  follows. 


Multiple-Correlation  Coefficient  R 


The  square  of  the  multiple-correlation  coefficient  is  defined  as  the  frac¬ 
tion  cf  the  total  variance  cf  Y  which  is  contributed  by  its  regression  upon  the 
variables  Xj,  X2.  — X„. 


n  m 

2 a,-  2 


o  1=1  7=1 

R2  = - 


2  v 


j- 1 


The  foregoing  is  obtained  ty  expanding  equation  D 1  and  grouping  into  total  sum 
of  squares  (denominator)  and  the  sum  cf  squares  due  to  regression  (numerator). 

A  value  cf  zero  gives  no  correlation  between  Y  and  X 1 ,  X2.  X„  where¬ 
as  a  value  cf  1  means  all  sample  points  lie  on  the  regression  plane. 


F  Ratio  Test 

The  F-test  determines  whether  the  form  of  the  regression  equation  is 
statistically  significant  ty  comparing  a  calculated  F  \alue  with  a  critical  F  fcr 
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(n)  aiid(m'M'l)  degrees  of  freedom  at  a  preselected  significance  level  of  alpha 

R2(m-n-I) 

"  (i'R2)  (n) 

If  the  calculated  value  of  F  is  greater  than  the  critical  value  of  F,  then  the  null 
hypothesis  that  all  cq  =  0  is  rejected  and  the  overall  regression  is  judged  to  be 
significant  for  die  alpha  significance  level. 

The  F'test  compares  the  am  of  die  squares  due  to  regression  with  the  sum 
cf  squares  due  to  error. 


Student  t  Distribution 


The  variable  t  has  the  Student  t  distribution. 


or 


X-fi  _  X-n 
s/yfn  s* 


where  - 

X  =  is  the  arithmetic  mean  cf  the  data  selected  for  a  random  sample 

of  size  n  r 

s  -  is  the  standard  deviation  cf  this  random  sample,  and  s  /J n  is  the 

standard  error,  s'  cf  X 

pi  =  is  die  arithmetic  mean  cf  all  the  values  composing  a  normal  popu  a- 
tion  that  has  a  standard  deviation  o. 

If  the  calculated  value  of  t  exceeds  the  critical  value  of  t  for'the  signi¬ 
ficance  level  selected  and  m-n-l  degrees  of  freedom,  it  can  be  said  there  is 
probability  a  that  the  actual  divergence  of  the  sample  mean  occurred  simply  ty 
chance . 

Ebr  n  greater  than  30,  the  nonnal  distribution  gives  a  sufficiently  precise 
approximation;  for  n£  30  the  t  distribution  should  be  used. 

The  level  of  significance  indicates  the  probability'  of  obtaining  a  value  of 
t  outside  the  range  of  i  t  (critical),  for  the  degrees  of  freedom  liom  1  to  30,  purely 
as  a  result  of  random  sampling  variation. 
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APPENDIX  E:  METHOD  OF  STEEPEST  DESCENT 


The  method  of  steepest  descent  is  an  optimization  procedure  that  will 
locate  an  extreme  value  (in  this  case  a  minimum)  of  a  criterion  function.  The 
method  can  be  extended  to  functions  with  constraints.  Changing  the  sign  of  the 
criterion  function  interchanges  the  roles  of  the  extreme  values.  For  example, 
the  minimum  of  F  (x,y ,z)  is  the  maximum  of  -  F(r,y,2). 

The  method  of  steepest  descent  uses  successive  approximations  to  find  an 
extreme  value  of  the  criterion  function.  Each  new  point  P ;  +  i  is  determined 
fiom  the  expression 


p«+i  =  (*,-+  i,yi+uzi+i)=(xi,yi,zJ  - M  — ,  - ,  — 

\  dx  dy  dz  i 


(*i,yi,zi) 


,  ,  .  .  dF  dF  dF\ 

In  this  expression  I  —  ,  — 

\dx  dy  dz  j 


is  the  gradient  of  F  (denoted  V  F)  and  is  a 


vector  in  the  direction  of  greatest  increase  of  F. 

That  the  gradient  is  in  the  direction  of  greatest  increase  can  be  seen 
horn  the  following  argument: 


dF  = 


dF 

dx 


dx 


dF  dF 

t  dy  t  —  dz 
U}  OZ 


=  VF-dP 


=  |VF||dP|cos  6 


This  expression  for  the  differential  or  increment  in  F  is  greatest  when  6  is  0; 
that  is,  when  the  gradient  and  the  increment  in  the  parameter  vector  are  co- 
directional. 

The  constant  A.  (A>0)  is  a  scalar  indicating  the  step  size  for  the  next  set 
cf  coordinates  in  the  direction  of  the  gradient.  The  use  of-AVp  indicates  that 
the  path  P1P2i  ■  ■  •.  Pn  leads  in  the  steepest  direction  to  a  maximum  of  -F; 
that  is,  to  a  minimum  of  F.  Thus,  the  name  “method  cf  steepest  descent.” 

Following  the  selection  of  a  new  point,  the  criterion  function  is  evaluated. 
“  F  (xi+  1,  Xf+  1 , 2 i  +  l)  <  F  (x,-  y ;  2,-)  and  the  variables  remain  within  present 

bounds,  the  point  P,+  [  becomes  the  new  point  of  departure  for  finding  P;+  2- 
If  not,  a  new  A  is  chosen.  There  are  several  ways  of  choosing  A,  giving  rise 
to  variations  of  the  method. 

When  the  magnitude  of  the  gradient  becomes  zero,  it  is.  concluded  that  F 
has  reached  a  minimum.  However,  three  things  can  happen  to  contravene  a  valid 
solution : 

1.  the  minimum  may  be  a  local  minimum. 

2.  the  point  may  be  on  a  ledge,  or 

3.  the  point  may  be  a  saddle  point. 

In  either  of  cases  2  or  3,  further  variation  of  the  variables  would  produce 
a  further  decrease  in  F. 
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The  method  of  steepest  descent  is  adaptable  to  problems  with  constraints 
cf  two  types: 


f(x,y,z)  =  a  (i) 

g(x,y,z)<b  (ii) 


A  constraint  of  type  (i)  can  be  written  f(x,  y,  z)  -a  =  0  so  that  z  is  defined  as  an 
implicit  function  of  x  and  y , 


2  =  <£  (x,y). 

Then  F(x,  y,  4>{x,y))  -  G(x,y),  and 


where 


VF  = 

\Bx  By 


dG  BF  BF  dz  ,  dG  BF 
dx  Bx  dz  Bx  dr  By 

Bz  Bz 

The  partial  derivatives —  and  —  are  evaluated  with  the 
Bx  dy 

the  differentiation  of  an  implicit  function  that  states  that 


dF  Bz 
St  By 

aid  of  a  theorem  on 


df 

dz  _  Bx  if  K  t  0,  and  similarly  for  ~ 

dll  “  "  d£  dz  dy 

Bz 


Each  constraint  equation  has  the  effect  cf  eliminating  one  variable  Iran  the 
problem.  In  the  case  discussed,  the  variable  eliminated  is  2,  since  the  function 
f  specifies  it  as  dependent  upon  x  and  y.  This  is  the  key'  to  the  problem  with 
constraints  of  type  (ii).  As  long  as  the  constraint  is  satisfied,  the  problem  is 
treated  as  one  with  no  constraints.  As  soon  as  the  constraint  is  violated  ty  some 
point  (xi,  y;,  z,-),  the  equality7  sign  is  assumed 'to  hold,  and  the  problem  is  treated 
as  one  with  a  type  (i)  constraint. 

The  method  of  steepest  descent  has  one  major  liability7;  namely,  the  local 
minimum  nearest  the  initial  point  (xj,  y  j,  zi)  will  be  found.  If  the  function  has 
more  than  one  local  minimum,  the  true  (global)  minimum  may7  be  missed.  The 
programmer  should  have  some  idea  cf  what  answer  to  expect  in  order  to  eliminate 
possible  spurious  answers  fcy  a  wise  choice  of  the  initial  values  of  x  ,  y,  and  z. 
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APPENDIX  F:  LIST  OF  SYMBOLS 

In  most  cases  in  this  report  the 
relationship  between  capital  and  lower¬ 
case  symbols  is:  F=  101og1Qf 

B  =  Bandwidth  relative  to  1  kc/s 
c  =  Number  of  channels 

Cf  =  Confidence  factor 

D  =  Frequency  shift  =  42.5  c/s 

=  Equivalent  vertically  polar¬ 
ized  ground-wave  root-mean- 
square  noise  field  strength 
in  dB  above  tyV/m  for  1-kc/s 
bandwidth 

F  =  Effective  receiver  noise 

figure 

Fa  =  Effective  antenna  noise 
figure 

Fc  -  Effective  receiver  antenna 
coupler  noise  figure 
fg  =  Center  frequency  cf  frequency 
ran8e  fbw 

fm  =  Band  width  of  low-pass  filter 

fr  -  Receiver  noise  figure 

fi  =  Transmission  line  noise  figure 

fx  =  Operating  frequency  in  x  units 

of  c/s 

fbw  ~  Frequency  range 

Gr  =  Receive  antenna  gain 

Gt  =  Transmit  antenna  gain 

hf  =  High  frequency 

K  =  Boltzmann  constant  1.38 

X  10-23  w/'K  c/s 
L p  =  Propagation  patli  loss 

m  =  Older  of  frequency  diversity 
N0  =  Noise  power  perT-c/s  band¬ 
width 

Pe  =  Bit  ciror  rate 

Pt  =  Average  transmitter  power 

Q  =  Quantity  cf  units 

R  =  STq/Nq  =  normalized  post¬ 

detection  S/N  ratio 
rct  =  Transmit  antenna  coupler 
VSWH 

RPRE=  Predetection  S/N  ratio 
Sr  =  Receiver  stability,  PPM 

St  =  Transmitter  stability,  PPM 

T  =  Temperature  (*K) 

T0  =  Baud  length  (pulse  length) 

vlf  =  Very  low  frequency 

a  =  'Standard  error 
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